Almost 50 years ago the first humans landed to the Moon. That was one small step for man, one giant leap for mankind. We in Aerokube team are not the astronauts. Our main activities on the Earth are in browser automation and we are doing our best to launch it to the stars. Today after almost two years of active discussions, development and testing we are happy to present our flagship enterprise Selenium solution called Moon working in Kubernetes cluster.
If you are following Selenium community — you should have heard about our open-source browser automation tool called Selenoid. Selenoid is actively propagating a new approach in browser automation — starting browser sessions in Docker containers, a new browser container for each test. This simple idea makes Selenium tests incredibly stable and reproducible. If your infrastructure lives on virtual machines in some cloud platform — then Selenoid and its companion Ggr, a load balancer to distribute load across multiple Selenoid instances, will be your best friends. Take a look at these articles if you wish to know more about Selenoid:
- Selenium: a new hope (part I)
- Selenium: a new hope (part II)
- Selenium: done in 60 seconds
- Selenium on Windows: revisited
- Selenium: easy as a pie
- Selenium: an Apple story
- Selenium: Growing Muscles
- Selenium: Clear as a Bell
Meanwhile the IT world is evolving fast. Four years ago containers and Docker were the bleeding edge deployment technologies used only by a few marginals. Today containers are an industry standard and Docker is an everyday tool for millions of developers. Containers can be approximately treated as a lightweight replacement of traditional virtual machines. This is why sooner or later the same scalability and reliability requirements should have been appeared for containers. I think most of you have already heard about container orchestration systems (or clusters of containers) — Docker Swarm, Apache Mesos or Kubernetes. All these tools are different solutions of the same problem — having a pool of servers be able to reliably distribute desired number of application containers across these hosts and provide various primitives to efficiently deploy and administer these containers. Very similar to mainstream cloud platforms, right? Two years ago when clusters of containers were experimental technologies we understood that this is where the world was going. We thoroughly compared Docker Swarm, Apache Mesos and Kubernetes and chose the latter one — Kubernetes (see an appendix at the end of this article for more details about the reasons).
Technically Kubernetes allows to deploy any application that can be packed to container and Selenium is not an exclusion. In theory you can easily settle Selenium server to Kubernetes and it will work. However doing so you lose most of Kubernetes advantages.
First of all fault-tolerance. To be fault tolerant an application should have two or more instances running on different hosts (and even better in different datacenters). Installing two or more copies of Selenium server to Kubernetes behind round-robin load balancer will not work. Every instance is stateful and stores a list of running Selenium sessions in memory (not shared between instances). Thus when you create a new session on the first instance and then try to work with it on the second instance — you will always get a “session not found” error response.
Next one — scalability. Just because of the same state being stored in memory known Selenium solutions scale hard in Kubernetes. Selenium Grid is defective because all the traffic is always passing through one entry point called hub. More load you add — more chances are that this hub will burst. Selenoid is also not suitable for Kubernetes. It is launching browser containers using Docker API and thus all you browsers will be started only on the same Kubernetes node where Selenoid is actually running. It makes no sense to use Selenoid with such limitation.
Finally — uninterrupted operation. Kubernetes comes with a cool feature called “rolling updates”. That means if your application has multiple instances — only one is updated at a time guaranteeing that no user request is lost or rejected. This works perfectly with general purpose web servers such as Nginx and still has no chance to work with known Selenium solutions. Every Selenium server instance restart means lost state, lost connections and thus broken tests.
These and many more other reasons led us to a simple conclusion — only a fully stateless solution will work reliably in Kubernetes and to be stateless this solution should be a first-class Kubernetes citizen.
Right now Moon supports most of the features needed to run and debug your tests:
- Running tests in ready to use browser images: Firefox 3.6+, Chrome 48.0+, Opera Blink 33.0+, Opera Presto 12.16.
- Video recording for every running session
- Logs recording for every running session
- Ability to automatically upload recorded log and video files to S3-compatible storage
It is better to see how it works once than to hear about it a thousand times. So here is a demonstration video:
How to try
- Clone our Github repository with ready to use Kubernetes YAML files:
$ git clone https://github.com/aerokube/moon-deploy.git
$ cd moon-deploy
- Apply the files:
$ kubectl apply -f moon.yaml
What about Licensing
Contrarily to Selenoid which is a 100% free open-source solution, Moon is a closed-source paid solution with commercial license. License is a subscription allowing to run a fixed number of parallel Selenium sessions. You can use Moon for free (without installing a license) if you have up to 4 parallel sessions.
Where to Get Help
We have several free support channels:
- Moon documentation: https://aerokube.com/moon/latest
- Our support email: email@example.com
- Our Telegram support channel: https://t.me/aerokube_moon
- Our Github issues page: https://github.com/aerokube/moon/issues
- Cool User Interface. Right now Moon is using Selenoid UI. However an enterprise solution should have more advanced user interface. We are already working on such new interface and it will be presented in next Moon versions.
- Advanced enterprise features. We would also like to add some advanved enterprise features like improved authentication allowing to use LDAP or ActiveDirectory as authentication provider.
If you have any proposals on how we could improve Moon — don’t hesitate to create an issue.
Bonus: Free License Key
We want to deliver a first-class polished Selenium solution and thus need more feedback from you. So here is how to get a free license key for desired number of parallel sessions:
- Open https://moon.aerokube.com/ and find Moon Evaluation License form.
- Fill your company name, an email address to send the license key and desired number of parallel sessions.
- Click on “Send me a key”.
You will then immediately receive a free license key valid until March, 1 2019 to the email specified.
Appendix: Why Kubernetes
Let’s return to three main container orchestration systems: Apache Mesos, Docker Swarm and Kubernetes and try to understand why did we choose the latter one. If you have already used our open-source tools such as Selenoid or Ggr you could have noticed that they are all implemented in Golang — a Google-backed programming language with powerful parallel computing and network processing tools. We consider Golang a perfect tool for devops applications and already have years of experience with it. This is why Apache Mesos implemented in C++ and requiring Java for Zookeeper is not for us. We now have two candidates — Docker Swarm and Kubernetes. Both are implemented in Golang and we need to dive into the details to get the difference.
First of all Kubernetes. Initially it was presented as an open-source implementation of principles used in Google internal orchestration system called Borg. However later it transformed to a big community maintained product supported by a big number of industry-known companies. Officially Kubernetes is hosted by the Cloud Native Computing Foundation and according to Github has more than 1500 contributors. Kubernetes from the first versions provided all important orchestration primitives such as pods, config maps, secrets, services and so on. Right now Kubernetes cluster can be launched in a few clicks in Google Cloud, AWS, Digital Ocean and Microsoft Azure — all the most popular clouds. Also is supports two possible types of containers — Docker and rkt and has a lot of other useful extension points.
Now Docker Swarm. Initially Swarm was an extension of Docker allowing to connect multiple hosts to each other and use them as a single cloud of containers. Initially Swarm even had exactly the same API (and thus worked with exactly the same
docker CLI commands) because the main idea was just the same - launch standalone Docker containers but in a cloud of hosts. Later however everything has changed. Docker team added primitives similar to Kubernetes: services, configs, secrets and so on. However this was still less powerful than Kubernetes. A year ago Docker team introduced Kubernetes support in their distributions and this is a sign that Kubernetes is a mainstream solution.
So knowing all this stuff — we definitely opt for Kubernetes.