Welcome to the Step-by-Step Guidance version of this project. Let's do this!
Want a complete demo of how to do this project, from start to finish? Check out our ๐ฌ walkthrough with Natasha ๐ฌ
๐ฃ If you're EVER stuck - ask the NextWork community. Students like you are already asking questions about this project.
In this project, we're going to learn how to use AWS Secrets Manager to securely store and retrieve secrets.
Imagine that you're about to build a web app that displays the names of the S3 buckets under your account. Now, for the web app to showcase your S3 buckets, it's gonna need AWS credentials. How should your web app get those credentials?
The first thing that might come to mind is to store the credentials in the web app's code itself. This is what your web app is going to do initially... but as you'll find out, this is not the greatest idea ๐
When credentials are hardcoded into your code, they can be easily accessed by anyone who has access to your code. And sharing code happens all the time - especially when you decide to showcase your project publicly in a GitHub portfolio!
This is why it's important to store credentials in a secure way. We're going to learn about a better way to store and retrieve these credentials using AWS Secrets Manager.
How in the world do we update an insecure web app to use AWS Secrets Manager?
In this step, we'll take a look at a simple web app that lists S3 buckets, copy the code to our computer, and set it up to hardcode some AWS credentials ๐ฌ
In this step, you're going to:
Access and Explore the Repository
๐ก What is GitHub?
GitHub is a platform where developers share and collaborate on code. It's like a social media for code!Public repositories on GitHub are great for open-source projects, but it also means that anyone can see the code. This is why it's super important to never put sensitive information, like passwords or AWS credentials, directly into your code, especially if it's in a public repository!
Understand the Application Code
๐ก What is app.py?
app.py is the main file containing the logic of our web app i.e. how the app will respond to user requests and interact with AWS services. In our case,app.py contains all the code that connects to your AWS account and lists the names of your S3 buckets.
๐ก Extra for Experts: How is app.py built?
app.py is built using FastAPI, a web framework for building APIs with Python. Think of it as a set of tools and rules that make it easier to build web apps that can talk to each other over the internet.FastAPI helps developers like us create APIs quickly and efficiently, with features like automatic data validation and interactive documentation.
Let's explore
๐ก What are import statements?
Import statements bring in external code libraries that our application needs to work. These libraries provide pre-built functionality so we don't have to write everything from scratch.
๐ก What does
import config do?
This line tellsapp.py to load settings from another file namedconfig.py . We'll check outconfig.py soon to see what kind of settings it holds.
Examine
๐ก What is
config.py ?
config.py is a file that contains the configuration settings for our web app. Think of it as your app's settings file.In our case,
config.py contains the AWS credentials for our web app -AWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY , andAWS_REGION . Without these credentials,app.py wouldn't be able to connect to your AWS account.
The Security Risks...
Notice how the
Exposing your AWS credentials publicly is a major security risk and is extremely unsafe for production applications. Once someone else gets access to your credentials, they can use them to access your AWS account, delete resources, steal data, and cause damage.
It's not just AWS credentials; database passwords, API keys for other services, and any kind of secret should never be directly embedded in your code.
To make this web app more secure, we'll make a copy of it in our local computer, and then we'll play around with how
Set Up Your Local Environment
We'll be using the terminal to make a copy of the GitHub repository's code:
๐ก What is a terminal?
A terminal is where you can interact with your computer (e.g. download files, run programs, install tools, etc.) using text commands. It's like you're telling your computer what to do in a chat, except you're using special commands instead of words!
From your terminal, let's head to your Documents folder, a common place to store project files. We'll clone the repository there.
๐ก What is a terminal?
A terminal is where you can interact with your computer (e.g. download files, run programs, install tools, etc.) using text commands. It's like you're telling your computer what to do in a chat, except you're using special commands instead of words!
From your Command Prompt, let's head to your Documents folder, a common place to store project files. We'll clone the repository there.
Clone the GitHub Repository
Now we'll clone the GitHub repository to your local machine. Cloning copies all the files from the GitHub repository to a new folder in your computer.
๐โโ๏ธ Where do I find the HTTPS-URL?
You can find the HTTPS-URL by heading back to the GitHub repository and clicking the green Code button.
๐โโ๏ธ Getting a 'Git not found' error?
That's totally okay! This just means you don't have Git installed on your computer.
- Head over to git-scm.com/downloads.
- Download and run the installer for your operating system.
- Once installed, restart your IDE and try again.
Still having issues?
Troubleshoot here or share your error with the NextWork community!
After cloning, we need to go inside the newly created folder that contains the web app's code.
In your terminal, type
Let's double-check that we're in the correct folder:
๐ก Extra for Experts: What do these files do?
- Dockerfile: Used for building the Docker image for the application - if you were to deploy this app, you would use this file to deploy it to container services like AWS ECR, Elastic Beanstalk or ECS.
- app.py: The main application file, containing the FastAPI code that defines the web app.
- config.py: Contains AWS credentials (currently hardcoded - this is what we'll fix!).
- index.html: The landing page for the web app, a simple HTML file.
- requirements.txt: Lists the Python libraries required by the application.
- static: Folder containing static files (like CSS) for the web app's styling.
๐ก Extra for Experts: What do these files do?
- Dockerfile: Used for building the Docker image for the application - if you were to deploy this app, you would use this file to deploy it to container services like AWS ECR, Elastic Beanstalk or ECS.
- app.py: The main application file, containing the FastAPI code that defines the web app.
- config.py: Contains AWS credentials (currently hardcoded - this is what we'll fix!).
- index.html: The landing page for the web app, a simple HTML file.
- requirements.txt: Lists the Python libraries required by the application.
- static: Folder containing static files (like CSS) for the web app's styling.
๐ก What are these credentials?
These are example access keys for AWS - we're going to imagine that these are real AWS credentials that would give someone access to your AWS account!We're using these test credentials as it's easier and safer to use than real credentials that would expose your account to security risks. Because these credentials are fake, putting them in
config.py won't actually let your web app work (you won't see this web app display your S3 buckets' names).
๐ก What are AWS access keys?
AWS access keys are like a username (Access Key ID) and password (Secret Access Key) that let applications to access your AWS account programmatically. When an application uses these keys, AWS treats the requests (e.g. creating or deleting resources, accessing data) as if they're coming directly from you.This is why securing your access keys is critical - if someone gets hold of them, they could use all your AWS services with your credentials, potentially racking up large bills or accessing sensitive data in your account.
๐โโ๏ธ Noooooo, I wanted to see the web app work!
If you want to see the web app work, you'll need to create your own AWS credentials and put them inconfig.py . Wanna do this? You can!We'll show you how to in the
๐ Secret Mission below.
Want to see this app in action? Here's your chance!
Let's put some skin in the game... your mission, should you choose to accept it, is to run this web app using your very own AWS credentials.
By doing this secret mission, you'll also get first-hand experience on why we need to secure our credentials. Later in this project, you'll need to take extra care that the real credentials you put in here don't get exposed publicly!
In this step, you're going to:
Sweet! Our web app now has AWS credentials to get it to work. But here's where things get interesting - we've got our credentials sitting right there in our code, and that's... well, not great ๐
Let's see what happens when we try to share this code on GitHub. To do this, we'll fork the original repository (i.e. make our own copy that's stored in GitHub), and then push our hardcoded credentials to the forked repository.
In this step, you're going to:
Woooo exciting - let's get you set up! Signing up is free and takes just 5 minutes.
๐ก What is Github?
GitHub is a place for engineers to store and share their code and projects online. It's called GitHub because it uses Git to manage your projects' version history.
Fork the Web App Code on GitHub
๐ก What is a fork (in GitHub)?
Forking a repository on GitHub is like making a personal copy of an existing repository, and editing and storing that copy in your own GitHub account. This is a great option if you'd like to showcase your version of the web app, and make changes to it without affecting the original repository.
๐ก What is forking vs. cloning?
When you fork a repository, you're making a copy of it in your GitHub account online. If your forked repository is public, anyone can see it!
- When you clone a repository, you're creating a local, offline copy. By default, no one can see it since it's stored on your computer.
Go back to the original GitHub repository page in your browser: https://github.com/NatNextWork1/nextwork-security-secretsmanager.
In the top right corner of the repository page, click the Fork button.
After forking is complete, you should be redirected to your forked repository.
The URL in your browser should now start with
Push Your Code to GitHub
Nice! With your forked repository available, you can now push your version of the web app. Let's make it public! We'll get back to our terminal to connect our edited code to the forked repository.
In case you haven't already, let's initialize Git in your local project directory. This tells Git to track the changes you make to the code.
Now, we need to connect your local repository to your forked repository on GitHub. To do this, we'll add a remote named "origin" that points to your fork's URL.
๐ก What is a remote in Git?
A remote is like an address book entry in Git that tells it where your code lives online. Instead of remembering the full URL every time, Git lets you give each location a nickname. In our case, we'll call it "origin" and it points to the URL of our forked repository on GitHub.P.S. You can actually name your saved URL anything, but "origin" is the standard that most developers use.
Let's verify that the remote "origin" was set up correctly.
In your terminal, type
Finally, let's push your local commits to your forked repository on GitHub. This command uploads your code to the "main" branch of your "origin" remote.
In your terminal, type
๐ก What does this command do?
This command uploads your local commits to your "origin" remote. The-u andmain tells Git to set make the main branch the upstream branch, so next time you push code, you can just rungit push and it'll automatically know where to push the code.
GitHub's Secret Scanning Block
Uh oh! You probably see an error message saying "Push cannot contain secrets". GitHub automatically scans your code for secrets and blocks the push if it detects any, like AWS credentials.
In our case, GitHub detected that you are trying to push code that contains an AWS Access Key ID and Secret Access Key in
Man, GitHub is so smart ๐ฎโ๐จ
Running into this error highlights the exact problem we're trying to solve in this project: how do you manage secrets securely so they're not exposed in your code? In this case, it's even stopped us from sharing our code publicly!
This is where AWS Secrets Manager comes in! In the next step, we'll learn how to use Secrets Manager to store your AWS credentials securely, plus retrieve them in your application without hardcoding them in
So, GitHub rightly blocked us from pushing our code with hardcoded credentials. This highlights exactly why we need a secure way to manage secrets! The solution is to use a dedicated service for managing secrets securely - AWS Secrets Manager.
In this step, we're going to use AWS Secrets Manager to store your AWS credentials securely. We'll create a new secret in Secrets Manager, and then update your
In this step, you're going to:
๐ก What is AWS Secrets Manager?
AWS Secrets Manager is a service that helps you securely store and manage secrets, such as database credentials, API keys, and other sensitive information. Think of it as a digital vault for your secrets.Secrets Manager encrypts your secrets and allows you to retrieve them programmatically in your applications, without hardcoding them in your code. It also offers features like secret rotation and auditing to further enhance security.
Why use Secrets Manager?
- Security: Secrets are encrypted at rest and in transit, and access is controlled through IAM policies.
- Centralized management: Manage all your secrets in one place.
- Rotation: Automate secret rotation to improve security posture.
- Auditing: Track access to secrets for compliance and security monitoring.
Now let's create a new secret to securely store our AWS credentials.
What types of secrets can Secrets Manager store?
Secrets Manager can store various types of secrets, including:
- Database credentials: Usernames and passwords for databases like MySQL, PostgreSQL, SQL Server, etc.
- API keys: API keys for accessing third-party services.
- OAuth tokens: Tokens for authentication and authorization.
- Any other sensitive information: You can store any text-based secret in Secrets Manager.
๐ก What is Encryption?
Encryption is like scrambling a message so that only someone with the right key can unscramble it and read it. In our case, Secrets Manager uses an encryption key to encrypt your secrets, so the access key ID and secret access key are scrambled and cannot be read by anyone without the right access.
What is AWS KMS?
AWS Key Management Service (KMS) is a service that helps you create and manage encryption keys securely in AWS. Secrets Manager uses KMS to encrypt your secrets, ensuring that they are protected at rest. You can also learn more in our project on using AWS KMS!
Now, we need to give our secret a name and description.
๐ก What do these optional settings do?
- Tags can help you organize and manage your secrets.
- Resource permissions allow you to control who and what can access this secret.
- Replicate secret allows you to replicate this secret to other AWS regions for disaster recovery and lower latency access from those regions.
๐ก What is Secret Rotation?
Secret rotation is the process of automatically changing your secrets on a regular schedule. This is a security best practice because it reduces the risk of compromised credentials. If a secret is compromised, it will only be valid for a limited time before it's automatically rotated.Secrets Manager can automatically rotate secrets for databases and other services. You can also configure custom rotation for other types of secrets.
๐ก When should you use secret rotation?
Secret rotation is best for high-risk credentials like database passwords, privileged API keys, and service account credentials. These types of secrets, if compromised, could give attackers access to sensitive data or critical systems, so regular rotation helps limit the damage window.However, you might want to skip rotation for keys (where automated rotation isn't supported), hardware security tokens, or legacy systems that can't handle credential updates without downtime.
Secret type: Other type of secret
Encryption key:
Secret name:
Description:
Secret replication:
Automatic rotation:
Alright, now comes the fun part! ๐จ We're going to transform our
In this step, you're going to:
Grab the Sample Code
We need the Python code snippet to integrate with our application. Let's copy the relevant portions.
In the Sample code section that appears, click on the Python 3 tab.
Now we'll update our
It's easier to do this in a code editor (like VS Code) instead of directly in the terminal, but the choice is yours!
Let's delete the old config.py file and create a new one with our secure code! The instructions for this depends on your operating system:
Let's get to know our brand new
The first two lines are import statements:
The core of the pasted code is the
๐ก Why are we adding new lines to the code?
Great question! So far, the code provided by Secrets Manager only help us with creating a function for retrieving the secret. But, config.py still has one more job left to do - it needs to retrieve the secret from the function we create (get_secret()) and assign the values to theAWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY , andAWS_REGION variables that ourapp.py code expects.
Now let's add the code to use the
If you've closed the file, open config.py again using
Use the down arrow to go to the end of the file.
Delete the placeholder comment line
Add the following code block: ```python return json.loads(secret)
credentials = get_secret()
AWS_ACCESS_KEY_ID = credentials.get("AWS_ACCESS_KEY_ID") AWS_SECRET_ACCESS_KEY = credentials.get("AWS_SECRET_ACCESS_KEY") AWS_REGION = credentials.get("AWS_REGION", boto3.session.Session().region_name or "us-east-2")
Open config.py in Notepad:
Go to the end of the file and add:
Since this piece of code uses a new package
๐ก What are these lines doing?
These lines are responsible for actually retrieving the credentials from the secret and assigning them to theAWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY , andAWS_REGION variables that ourapp.py code expects:
secret_json = json.loads(get_secret()) : This line calls theget_secret() function to retrieve the secret from Secrets Manager. The secret is returned as a JSON string, so we usejson.loads() to parse it into a Python dictionary.AWS_ACCESS_KEY_ID = secret_json['AWS_ACCESS_KEY_ID'] : This line extracts the value of theAWS_ACCESS_KEY_ID key from thesecret_json dictionary and assigns it to theAWS_ACCESS_KEY_ID variable.AWS_SECRET_ACCESS_KEY = secret_json['AWS_SECRET_ACCESS_KEY'] : This line does the same for theAWS_SECRET_ACCESS_KEY .AWS_REGION = region_name : This line sets theAWS_REGION variable to theregion_name we defined earlier.
Woohoooo! Now our application is configured to retrieve AWS credentials securely from Secrets Manager - instead of hardcoding them ๐ฎโ๐จ
Let's share our newly secured code with the world! ๐ Remember how GitHub's secret scanning blocked our push earlier? Now that we've properly moved our credentials to AWS Secrets Manager, we can confidently push our code without exposing any sensitive information.
In this step, you're going to:
Save, Commit and Push Your Changes
Make sure you're back in your terminal.
Stage all the changes you've made in your local project by using
In your terminal, type
Nice! You should see output showing you that files have been changed and committed.
Finally, let's push your local commits to your forked repository on GitHub. Run
Oh no! We still get the same error in the terminal!
๐ก Why is this happening?
Turns out, simply editing theconfig.py file is not enough for security best practices ๐คฆโโ๏ธThe hardcoded credentials still live in your commit history, which is a record of all commits you've made to your repository. In our case, because we made an older commit in
๐ด Step #2 with the hardcoded credentials, someone else could still go through the commit history to find the credentials.To solve this, we need to rewrite the history to completely remove the commit that had the hardcoded credentials.
Remove Hardcoded Credentials from Commit History
We'll have to use use a special command called
Once you've found the commit ID, take note of the first 7 digits of that ID. This is going to be important! You can even copy it and paste it in a random text file to reference it later.
Once you've identified the commit, run the following command in your terminal:
๐ก What is
git rebase -i --root ?
git rebase starts a rebase session, which means rewriting your commit history.The
-i flag makes it interactive, meaning you can edit the list of commits to be rebased.
--root tells us that the rebase should start from the very first commit in your repository's history. This is useful when you need to modify commits from the beginning of your project.
๐ก What is
drop ?
Drop in the interactive rebase editor tells Git to completely remove a commit from your history as if it never existed.This is how we'll erase the commit containing the sensitive credentials. Use this command with caution, as rewriting history can cause issues like merge conflicts (which you'll see in just a second).
Git will now start the rebase, so it will remove the commit you marked for deletion from your commit history.
Oh wait! Looks like the rebase is not so simple after all. You might run into merge conflicts during the rebase.
๐ก What are merge conflicts?
Merge conflicts happen when you've changed the same lines of code in different commits.In our case, the commit we removed had the hardcoded credentials, and the next commit we did also changed the
config.py file. So Git's now confused - if you remove the commit with the hardcoded credentials, should it delete the change you made after that commit too?If a merge conflict occurs, Git will pause the rebase and ask you to resolve the conflicts first. To resolve these conflicts, we'll need to manually edit the conflicting files to resolve these conflicts.
Resolve Merge Conflicts
๐ก What are these merge conflict markers?
When Git detects conflicting changes in a file, it adds special markers to show you exactly where the conflicts are:
<<<<<<< HEAD marks the beginning of your current version======= separates the two conflicting versions>>>>>>> feature-branch marks the end of the incoming changesTo resolve the conflict, you need to choose which version to keep and remove all these markers.
๐ก What are these merge conflict markers?
When Git detects conflicting changes in a file, it adds special markers to show you exactly where the conflicts are:
<<<<<<< HEAD marks the beginning of your current version======= separates the two conflicting versions>>>>>>> feature-branch marks the end of the incoming changesTo resolve the conflict, you need to choose which version to keep and remove all these markers.
๐ก What are these merge conflict markers?
When Git detects conflicting changes in a file, it adds special markers to show you exactly where the conflicts are:
<<<<<<< HEAD marks the beginning of your current version======= separates the two conflicting versions>>>>>>> feature-branch marks the end of the incoming changesTo resolve the conflict, you need to choose which version to keep and remove all these markers.
Great job! You've just removed some prettyyyy sensitive data from your Git commit history. Now, let's verify that the AWS credentials can't be found in your public repository!
Verify Your GitHub Repository
Let's check on GitHub that the commit history is updated and your
๐โโ๏ธ What are the other commits in the history?
The other commits are the ones made by the NextWork team when preparing thisconfig.py for the project!Notice how easy it is to click through the commit history and see old versions of the file. As a little Easter egg, you might even notice that one of the older versions contains AWS credentials - these are not your credentials, but credentials that were entered in the original repository before you forked it!
This just goes to show how important it is to remove sensitive data from your commit history - even if it's no longer the current version of the file, someone could still get to your secrets over the history of the repository!
Nice work! By taking the time to secure your code and remove sensitive data from your commit history, you've made development easier, not harder. Instead of worrying about accidentally exposing credentials, you can freely share and collaborate on your code!
Now that we've secured and deployed our web app, it's important to clean up the resources we created to avoid incurring unnecessary costs.
Resources to delete:
Delete the Forked GitHub repository.
Delete the Secrets Manager secret.
Delete the local repository.
Delete the resources from the Secret Mission (optional ๐).
๐ก Why delete the GitHub fork?
Even though we removed the hardcoded credentials from our code, it's good practice to delete the forked repository if you don't plan to use it further. This helps keep your GitHub account organized and removes any potential security concerns from old code versions that might still exist in the repository's commit history.
Go to the Secrets Manager console.
Select Secrets in the left navigation pane.
Select the secret you created for this project, named
Click the Actions dropdown button.
Select Delete secret.
In the confirmation dialog box, type
Click Schedule deletion.
Wait for the secret deletion process to complete. Secrets Manager typically waits for a recovery window (e.g., 7 days) before permanently deleting the secret. You can choose to force delete it immediately if needed.
๐ก Why delete Secrets Manager secret?
While Secrets Manager itself does not incur significant costs for just storing secrets, it's a good security practice to delete secrets that are no longer needed. This reduces the risk of accidental access or misuse of the secrets in the future.
We created a few extra resources for this project's secret mission! Let's delete them one by one
IAM Access Key
* Go to the
๐ก Why delete IAM Access Key?
If you created a new IAM Access Key specifically for this project, it's a good security practice to delete it when you are finished with the project. This reduces the risk of the Access Key being compromised or misused in the future. It's recommended to only keep Access Keys that are actively being used and delete any unused ones.
S3 bucket
If you created a new S3 bucket for this project, you can delete it now:
Nice work! ๐ You've successfully learned how to protect sensitive credentials using AWS Secrets Manager!
You've learned how to:
๐ p.s. Does it say "Still tasks to complete!" at the bottom of the screen?
This means you still have screenshots left to upload, or questions left to answer!
- Press Ctrl+F (Windows) or Command+F (Mac) on your keyboard.
- Search for the text Return to later.
- Jump straight to your incomplete tasks!
- ๐โโ๏ธ Still stuck? Ask the community!