Principal Architect, building DS pipelines and architectures to process and analyze big amounts of data.
From Script to Pipeline: Long Road and Shortcuts
Time & Date
9:00, 24 March
There is a long path between a Python script and a production solution. How to start some new open source project? How can you build it into a proper package and make this complex CI/CD pipeline to pick it up and deploy automatically on some staging environment?
I propose us to go from a basic Python script to a package, then from package to a CI/CD project on Jenkins (let’s deploy it automatically as well). And all this goes further up to a deployed version somewhere that you can look at not so long after you’ve merged your last Pull Request on GitHub.
This workshop is a tutorial giving some illustration on what IaaC really is, and how you can automate all different steps on this path to your own pipeline.
Part 1: Intro to Python packages
- Typical structure of a good Python project
- What do you need to convert your Python code into package
- How to manage Python packages inside your infrastructure
Part 2: Jenkins deployment with Ansible
- Configuration management and Roles
- Configuring Jenkins Role to our needs
- Security and credentials management for GitHub integration
- Deploying ready-to-work Jenkins with just one command
Part 3: Jenkins jobs and IaC
- Using Groovy + Gradle to write job DSL
- Production pipelines for Python projects
- How to further organize your workflow
The main goal of this workshop is to help participants to go through the steps that convert a basic Python script into a production package and setting a full-featured CI/CD pipeline to maintain its development cycle.
The target audience includes people who are interested in IaC concept as well as building CI/CD pipelines from scratch. Python is used as an example of a popular programming language of choice, but the rest of the workshop is applicable to other kinds of projects as well.
– Python 3.5+ locally on a machine that will be used for running deployment scripts
– Cloud-based (or local) VM preferably with Ubuntu 16.04+ (CentOS 7 may also work) with Python 3.5+ and Java 8 installed (should work with both OpenJDK/Oracle)
– Technical knowledge:
– Basic programming in Python or some other programming language
– Basic SSH/shell usage