Welcome to the final part of this 4 part mini series on using Jenkins as a CI/CD build system for dotnet core projects.
So far we've seen how to set the server up, how to get it to build your app, in this final part we'll see how to get Jenkins to deploy and perform a basic "I'm running test" of your application to tell if your build is alive or not.
It's all about the build
At the end of part 3 we left our app built but dangling, with nowhere to go. Since our goal is not only to make sure our project builds, but to also make sure it runs, we need to ask Jenkins to place our app somewhere, and to set our build server up so that it deploys our service and add's it to the OS.
Just as with building our app, it's the build scripts job to also make this part of the puzzle happen too.
Sorry guv, you need permission to do that…
Before we can do any of this though, we need to talk about permissions.
Those of you who are used to deploying on windows may never have had to deal with Linux/Unix style permissions before, so a quick refresher is in order. If you know your way round the linux permissions model then you can skip this part.
Linux enforces run time permissions on applications more stringently than windows does. For example, you cannot just right click anything you want to and "Run as Administrator", and while you can login as the "root" user (Linux's answer to the windows Administrator account) doing so is made extremely difficult and everything possible is done to persuade you not to.
Most applications that are not critical to the running of the system as a whole are therefore forced to run using their own groups and user settings.
It's much the same when you log in with your user account. If you look at any applications you run as a user in the system process monitor, you'll see that they run as your user in your user's main group.
Many of the more common applications (Such as web servers) generally run using a common name/group, which in most cases is often "www-data", however in order to attach to the operating system, they "impersonate root" first, then switch to a less privileged user.
This means that in order to boot strap a service at boot time, it needs to be installed in the system using "root" privileges something which Jenkins does not have permission to do.
When you install a Linux system the first time round, you get asked for the "First User" to add to the system. It's important that you don't lose this user's information, as this user by default is the ONLY user on the system that will have the ability to use "SUDO". Sudo is the ability to elevate a user account to use commands with root privileges, a bit like windows asking if it can elevate the current user to admin in order to perform a system task.
Unlike windows however, sudo is ONLY attached to those users the system owner deems need that level of access, and secondly, a password (Generally that users normal password) is still requested any time a sudo based action is requested.
Our goal in getting Jenkins (and it's associated 'Jenkins' user and group) to deploy our app is therefore to give Jenkins access to "sudo" and to have it give our application a user/group of "www-data"
Bring on the web server
If you've read the official Microsoft Advice on hosting dotnet core projects, you'll know that it's not advised to have your service listening directly on the normal web server port (Port 80) taking requests.
Instead, what MS advise you to do is to place a server designed to take the onslaught of the public web, in front of your application, and make your application available to this web server using a "localhost" or internal network connection.
For most Linux users this means either Apache or Nginx , there are others but the two mentioned are by far the most common choices.
For us, where going to use "nginx", which like any other application install is a simple apt command line to install it, just as we did in the other parts of this guide, to install Jenkins and the tools we needed to build the server.
Install the web server by typing
sudo apt install nginx
At your SSH or local command line (See part 1 for details on this aspect)
Notice the use of "sudo" here?
This is the exact sudo privilege I was talking about a few paragraphs ago, it's allowing you to temporarily have root privileges in order to install the web server software and will last ONLY for the duration it takes the "apt" command to run, as soon as "apt" is finished and exits the extended privileges will be removed and the command line will go back to the normal user it was started with.
Once apt finishes installing nginx, you should be able to point your browser at the IP address of your Jenkins server on port 80, and see the default black text on white "Welcome to Nginx" page.
I'm not going to screen shot it however as we'll be configuring our system to ignore this, and route every request coming in to this server to our dotnet core application.
Once you're happy that nginx is listening and serving requests, type
Sudo mc
At your command line to start midnight commander as a root user, then browse to the folder "/etc/nginx/sites-available". In this folder you should find a file called "default", press F4 on this file to edit it, you should be greeted with something that looks similar to the following:
Fig: Default Nginx config ready for editing
Since we won't be using most of this, repeatedly press F8 in the editor until all the lines of the file have been removed, then paste or type in the following configuration:
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm;
server_name _;
location / {
proxy_pass_header Authorization;
proxy_pass http://localhost:5000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
client_max_body_size 0;
proxy_read_timeout 36000s;
proxy_redirect off;
access_log /var/www/apps/logs/st_internalui.access.log;
error_log /var/www/apps/logs/st_internalui.error.log;
}
}
Please make sure you change the highlighted part in the code above to point to your application. If your app is using the default dotnet core kestrel settings and is on the same server as the one you're running the web-server on, then localhost on port 5000 will work fine. If you've changed it in any way however, you'll need to change the config to suit, if you don't get that setting correct, then Nginx won't be able to talk to your application and exchange data with it.
Press F10 to exit and save the file, and return back to the file selector.
While we have MC open and in sudo mode, we'll make the other changes that are needed to deploy and run our application, ready to deploy.
Browse to the folder "/etc/system/system" then press ctrl+o to switch to command entry mode and enter
touch myapp.service
This will create an empty file for your application to be launched.
Press ctrl+o to switch back to MC, you should now see your file in the file list. Use your arrow keys to move to the file and press F4 to edit it.
Fig 2: Empty editor ready for our service config
In the editor, paste or type in the following file definition:
[Unit]
Description=My Application
[Service]
Type=simple
User=www-data
Group=www-data
NoNewPrivileges=true
WorkingDirectory=/var/www/apps/myapp
ExecStart=/var/www/apps/myapp/myapp
Restart=always
StandardOutput=append:/var/www/apps/logs/myapp.stdout.log
StandardError=append:/var/www/apps/logs/myapp.stderr.log
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=myapp
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_URLS=http://localhost:5000/
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
[Install]
WantedBy=multi-user.target
Again as previous, pay close attention to the highlighted sections and make sure you change those as needed for your system.
The description is a human readable text that will appear in system notifications and system emails relating to your application.
The working directory and exec start parameters MUST point to the folder where your app has been copied too AND the output file produced by dotnet core that is called to run it, depending on how you deploy this may be a direct exe name or it may be a full dotnet command line. In this guide we'll be using the direct exe name as outputted by the "dotnet build; dotnet publish" process.
The user and group parameters MUST be the user and group your app will be run as, just as we discussed earlier on, which in our case with Nginx will most likely be "www-data"
Once you have this config saved by pressing F10 to exit the editor and saving your file, you then need to register that file with the OS system app manager "systemd", press ctrl+o and type:
systemctl enable myapp
At your command line to enable the app to start at boot time, we can't attempt to "start" the service yet as we don't have a set of files deployed from our build.
Note also that if you called your service something other than "myapp.service" then you need to substitute the "myapp" part in that last command for the correct name.
We still have one more change to make to the OS before we move on, so from the MC file manager screen browse to the "/etc/sudoers.d" and create a new file called "Jenkins" using the following command line command:
touch Jenkins
Press ctrl+o to switch back to MC, edit this new file and make sure it contains the following 2 lines:
%jenkins ALL=(ALL) NOPASSWD: /bin/systemctl start myapp
%jenkins ALL=(ALL) NOPASSWD: /bin/systemctl stop myapp
It should look similar to the following:
Fig 3: Our "SUDO" definition
Remember to change the highlighted parts to match the name you gave to your ".service" file previously when you created the service definition to start your app.
What these 2 lines do is they instruct the "sudo" system to allow Jenkins to call the two specified "start" and "stop" commands, as a privileged root user without needing a password.
While this may seem like an insecure thing to do, it's a necessary evil, as Jenkins runs headless, so there will be no once watching and available to type a password in.
Notice however, we are explicit in the exact command and its parameters, we could for example specify "/bin/systemctl *" on one line, and that would allow Jenkins the ability to start/stop any service we wanted, BUT it would also give those permissions to a bad actor that might break into the system using a low privileged user such as Jenkins as an attack vector, so we DON'T use wildcards and make it so that the exact commands Jenkins needs, and ONLY those commands are added to Jenkins own SUDO definition.
Making a copy of the files
With everything configured, stay in MC and navigate to "/var/www". The WWW folder is the default folder where Nginx creates its default website files.
Create a folder called "apps/myapp" to store your dotnet application in, then select and press enter on that folder to navigate into it. Press tab to swap to the opposite file pane, then navigate that side of MC to the "/var/lib/Jenkins/workspace" folder. Inside this folder you should find a folder named the same name as your project, navigate into this folder so you can see the file list, you should see something similar to the following:
Fig 4: Our project on the right, our app folder on the left
Notice in my image I have a folder called "published" where I've redirected my compile output to, in yours you most likely will have a normal "bin" folder, just as you would have in your visual studio project folder on windows.
Navigate into the correct folder, either published or bin as needed to find the output from your build process.
With the file selector in the right window, press the '+' key on your number pad, then enter '*' in the dialog the opens to select all the files, then press F5 to copy them to your app folder.
It should look something similar to:
Fig 5: Our files are copying
Once your build files have copied to your "/var/www/apps/myapp" folder, press the tab key to switch back to that side of MC, then press '+' and enter '*' to select all your files, then press 'ctrl+x' followed by 'o' to open the user & group editor.
Fig 6: The user and group editor
Set the user and group to 'www-data' for our files, then select and press enter on set or set-all to set our file selection with the correct user and group settings.
Navigate back one level to the 'myapp' folder, and do the same again to set that to 'www-data' only this time after you set the user and group repeat the command, but this time use 'ctrl+x' followed by 'c' to open the permissions editor.
Make sure the folder permissions are set as seen in figure 7 below:
Fig 7: Setting the app folder permissions for jenkins
We set these permissions on the folder so that Jenkins is allowed to write to the folder as www-data in order to deploy our app to nginx.
Press 'ctrl+o' to switch to command line mode and enter the following commands
usermod –aG www-data Jenkins
usermod –aG Jenkins sudo
These two commands will assign Jenkins to the www-data user group, and will assign sudo privileges to the Jenkins user so it's allowed to run the sudo commands we set up earlier.
You should now be able to type
service start myapp
At your command line, and watch your service start up, and run under Nginx, and pointing your browser at the IP address of your server now, should show your app instead of the Nginx default page.
Back to Jenkins we go
With all the OS setup behind us, the final part of all this is to add the extra commands needed so that Jenkins can take charge, copy and start/stop our service as needed.
In the build section we created last month, add an extra line to publish your application (This will create the publish folder you saw earlier when copying the files), your standard build rule should look like the following:
Fig 8: Add a publish rule to your build
The publish rule should be something like:
dotnet publish –c Release –o published
Once this has been changed, add a NEW "Conditional Step (single)" to your build by using the "Add build step button, and set it up as seen in "Fig 9" below:
Fig 9: Add a new conditional build step
The execute shell command window should contain the following script code:
sudo systemctl stop myapp
rm -rf /var/www/apps/myapp/*
mv "${WORKSPACE}/published/"* /var/www/apps/myapp
sudo systemctl start myapp
echo sleeping for 10 seconds to allow service to load and stabilize
sleep 10s
This script will use the powers we've granted Jenkins to stop the current running version of your app, delete the old copy from the web server folder, copy over the newly built version, restart the service then wait 10 seconds for it to stabilize before moving on to the next step.
Note that we configure this step to ONLY run if the previous part of the build was successful, if our solution failed to build or dotnet did not run correctly, the build will be stopped and no files will will be deployed. We do this so that if our build fails we don't overwrite the current running copy with a failed version.
Add one more build step using the add build step menu, this time add an "HTTP Request" step.
If you recall back in part 2 when we first installed Jenkins, I told you to make sure you added the HTTP request plugin. That plugin is responsible for this step, so if you can't find this step in your menu, go back to your plugin setup and make sure you included it.
The setup should ignore SSL errors, perform a GET request and talk direct to the dotnet service running on localhost, its setup should look something like this:
Fig 10: Our HTTP request step checks to make sure our app is running
The HTTP request is used to make sure that our app started correctly, and will fail the build if it did not. If your app uses something like ASP.NET health checks then here would be a good idea to call the health check endpoint rather than the web root.
All good stories must come to a close...
And that as we say in the business is "Game Over".
You should now have a versatile little setup that will allow you a lot of flexibility in deploying dotnet core web applications within your network.
There's tons of stuff you can experiment with here, from SSH based commands to get Jenkins to actually SSH into other servers, to copying and deploying via windows shares and samba services.
You can also send emails, update slack channels, create build badges and so many other things to show project status.
In my builds I usually write a build tag with the build number in, back to the git source control repo, so that if I need to roll back to a specific successful build, I can.
Jenkins can be deployed on AWS, or into a linode virtual machine so you can perform your builds in the cloud, you can build to a staging server, then have someone use a chained Jenkins setup to auto deploy an approved build out to production.
I hope you enjoyed this little 4 part mini series, and learned something new from it.
If you want to reach out to me you can usually find me hovering around on twitter as "shawty_ds", or doing weird things with dotnet code on github as "shawty".
Just don't push your new digital butler too hard… even digital butlers have rights you know ;-)