Time travel for containers

Software development in the financial sector also involves extensive testing — especially with regard to how the software handles time-related issues. Are the accounts calculated correctly at the end of the year? What happens on a leap day?

In the “good old days,” all you had to do was adjust the server's clock to test the software's behavior. In the age of containers, that's no longer possible — systems are far more complex, and tampering with server clocks can throw the cluster out of balance, making operation and testing impossible; it can even destroy the cluster.

Using the open-source software “libfaketime,” containers can be “shifted through time” without requiring any changes to the application, the code, or the image. It is even possible to run tests with hundreds of containers in parallel and “at different points in time.”

At the data center of a major German financial services provider, we successfully integrated a time-stamping process into the CI/CD environment using the open source Linux library libfaketime.

Quick takeaways

• Using the open-source Linux library “libfaketime,” you can test how software behaves when faced with time-related issues.

• There’s no need to modify the applicaion, the code or the image.

• All that’s needed to be done is to inject the library by using an init container and add two environment variables to control the time inside the container.

• The custom-built Init container, developed in collaboration with the testing team and using the customer’s infrastructure, integrates seamlessly into the customer’s IT environment, providing them with autonomy and expertise so they can adapt it to new challenges at any time.

How it works

To manipulate time within the container, the “libfaketime” library is inserted between the application and the kernel, where it can intercept all system calls related to time. The core of the whole process lies in the LD_PRELOAD environment variable, which ensures that the system loads the defined libraries before all others. This is nothing new; it is a well-established technique in Linux. But in the modern world of containers, this environment variable takes on a new significance.

With LD_PRELOAD, it is not only possible to link libfaketime into the process, but also to do so without having to modify the application, the code, or the image. All that is needed is an init container that writes the library to a shared volume, from which the application container preloads it. That's all there is to it.

That sounds simple, but the challenges usually lie not in the technology, but in the logistics. In a Kubernetes (or OpenShift) environment, applications are generally deployed via YAML files or—even more standardized—via Helm charts. The deployment process is managed and controlled by a CI/CD system such as Jenkins, GoCD, Bamboo, or similar solutions; some companies, like our client, even use their own software to handle these tasks.

This is where your journey begins: The deployment description must be expanded to include details about init containers, shared volumes, and several environment variables, such as the “FAKETIME” variable to define the “target time.” Additionally, a switch is required to turn time travel on and off. The teams involved need to be brought to the table to discuss and adjust processes. That is your job. When it comes to technical details, such as the init container, etc., katalytic IT is happy to assist you.

Why libfaketime?

At the customer’s site, a team of testers responsible for testing software on Windows, Unix, and Linux servers was confronted with a new paradigm in software hosting: Kubernetes. When it came to testing time-related issues, it quickly became clear that containers don’t have a clock that can be tweaked. Containers don’t have one, and changing the time on the nodes would immediately throw the cluster into disarray. Therefore, a software solution was needed.

There are several solutions on the market that address this challenge. But all of them are proprietary, difficult to manage, hard or impossible to adapt to customer-specific requirements, and expensive. The software’s Docker images often cannot or are not allowed to be modified, and the mechanisms within the container would be a “third-party black box”—a detail that could pose a compliance issue in the financial sector.

Instead, they opted for an open-source alternative—and that’s where katalytic IT came into play. We developed an init container that leverages the customer’s existing infrastructure and is tailored precisely to their needs. Development and testing were integrated into the existing CI/CD landscape, while the source code for the library, the image, and the pipelines was managed in the central version control system—transparent, traceable, and verifiable at any time.

Perhaps the entire process took longer than it would have if they had purchased third-party software. But the investment paid off in the long run: not only because there were no monthly licensing costs, but also through the knowledge gained by the team and the confidence in their own product, since they know exactly what it does and that it can be adapted to new challenges at any time. The team has earned autonomy here — a priceless asset.