<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="z1g1.net/feed.xml" rel="self" type="application/atom+xml" /><link href="z1g1.net/" rel="alternate" type="text/html" /><updated>2026-06-18T19:47:49+00:00</updated><id>z1g1.net/feed.xml</id><title type="html">z1g1.net</title><subtitle>Zack Glick personal website</subtitle><entry><title type="html">How to Stop Using Microsoft Account to Login to Atlassian</title><link href="z1g1.net/daily%20learning/2024/10/04/unlink-atlassian-account.html" rel="alternate" type="text/html" title="How to Stop Using Microsoft Account to Login to Atlassian" /><published>2024-10-04T00:00:00+00:00</published><updated>2024-10-04T00:00:00+00:00</updated><id>z1g1.net/daily%20learning/2024/10/04/unlink-atlassian-account</id><content type="html" xml:base="z1g1.net/daily%20learning/2024/10/04/unlink-atlassian-account.html"><![CDATA[<p>If you are logging into Atlassian with your Microsoft/Google account and want to set a password you need to <a href="https://support.atlassian.com/atlassian-account/docs/manage-linked-third-party-accounts/">unlink it</a>. To do so you need to change your email address.</p>

<p>However, if you are the organization admin you have to follow <a href="https://community.atlassian.com/t5/Confluence-questions/How-do-I-change-the-email-address-of-organization-administrator/qaq-p/1638975#:~:text=To%20change%20your%20email%20address,the%20Administration%20%3E%20User%20management%20section.">specific instructions</a> to change your email. You need to make sure that you use <code class="language-plaintext highlighter-rouge">Directory &gt; Managed Accounts</code> not <code class="language-plaintext highlighter-rouge">Administration &gt; User management</code>.</p>

<ol>
  <li>Once you are in the <code class="language-plaintext highlighter-rouge">Managed accounts</code>` screen you can change your email which will unlink your account</li>
  <li>Trigger a password reset and set a password via the link sent to your email</li>
  <li>Set the new password</li>
  <li>On the <code class="language-plaintext highlighter-rouge">Managed accounts</code> screen you can change the email back</li>
</ol>]]></content><author><name></name></author><category term="Daily learning" /><summary type="html"><![CDATA[If you are logging into Atlassian with your Microsoft/Google account and want to set a password you need to unlink it. To do so you need to change your email address.]]></summary></entry><entry><title type="html">My appearance on the Modern Cyber Podcast</title><link href="z1g1.net/media%20appearance/2024/08/29/modern-cyber-podcast.html" rel="alternate" type="text/html" title="My appearance on the Modern Cyber Podcast" /><published>2024-08-29T00:00:00+00:00</published><updated>2024-08-29T00:00:00+00:00</updated><id>z1g1.net/media%20appearance/2024/08/29/modern-cyber-podcast</id><content type="html" xml:base="z1g1.net/media%20appearance/2024/08/29/modern-cyber-podcast.html"><![CDATA[<p>My appearance on <a href="https://www.youtube.com/watch?v=C9SS50wz9CM">Episode 26 of the Modern Cyber podcast</a> with Jeremy Snyder of <a href="https://www.firetail.io/">FireTail.io</a> was recorded at the <a href="https://fwdcloudsec.org/">fwd:cloudsec US 2024 conference</a></p>]]></content><author><name></name></author><category term="Media Appearance" /><summary type="html"><![CDATA[My appearance on Episode 26 of the Modern Cyber podcast with Jeremy Snyder of FireTail.io was recorded at the fwd:cloudsec US 2024 conference]]></summary></entry><entry><title type="html">How to Create a Hardened Cloudflare Account</title><link href="z1g1.net/zatik/2024/04/22/hardened-cloudflare-account.html" rel="alternate" type="text/html" title="How to Create a Hardened Cloudflare Account" /><published>2024-04-22T00:00:00+00:00</published><updated>2024-04-22T00:00:00+00:00</updated><id>z1g1.net/zatik/2024/04/22/hardened-cloudflare-account</id><content type="html" xml:base="z1g1.net/zatik/2024/04/22/hardened-cloudflare-account.html"><![CDATA[<p>Original post on <a href="https://www.zatik.io/blog/how-to-create-a-hardened-cloudflare-account">Zatik’s blog</a></p>

<blockquote>
  <blockquote>
    <p>Zatik Security moved all of our domains and DNS infrastructure onto the Cloudflare platform in early 2024. We made this decision after completing a review of 5 commonly used domain registrars security features. We selected the Cloudflare platform based on the security features available on their intro tier without requiring an upgrade to the enterprise tier. These features are strong two factor authentication,  support for multiple users with role-based access controls, in-account audit trails, and the ability to log out other users. Support for these features is included in the intro tier, but still needs to be configured. Follow along below as we create and harden a Cloudflare account in April 2024. If you just want the step by step without the narrative you can skip right there. 
you can have one category listed</p>
  </blockquote>
</blockquote>

<blockquote>
  <h2 id="root-account-creation">Root Account Creation</h2>
</blockquote>

<blockquote>
  <p>The setup for Zatik Security on Cloudflare is a root account which will be the parent of all other accounts. This root account will only be used during the creation and setup of the account. You should never use the root account as your day-to-day account. Any staff who need access to Cloudflare will log in with child accounts which are invited to the root account. Their invitations are scoped to only have the minimum permissions required to complete their tasks. For example, a member of the finance team could have an account which is only able to manage payment in Cloudflare but not alter any technical settings. Or a contractor hired to work on a marketing website can be granted access to manage the DNS records of that website only.</p>
</blockquote>

<blockquote>
  <p>Since employees have their own accounts for day-to-day activities you can lock down access to the root account to a very limited number of individuals at your company. Reducing the use of this account and increasing friction at login will reduce the probability that the account will be compromised. The root account is reserved for a “break glass” scenario where a child account with administrative privileges is compromised and an attacker uses this elevated privilege to revoke all other access to the account. Since this attacker-controlled account is a child of the root account, it cannot revoke the root account’s access. You can then use the root account to remove the attacker-controlled account from your Cloudflare account, thereby containing the attack.</p>
</blockquote>

<blockquote>
  <p>Here’s how we set up our accounts on Cloudflare: First, create an email address that is not associated with a specific user at your company, such as, admin@example.com. Use this email address to create a new account at Cloudflare. Store the password in the password manager that your company makes use of.</p>
</blockquote>

<blockquote>
  <p>After creating the account, ensure to verify the email address on the account profile page you verify this email address. Finally, on the Configuration Page (https://dash.cloudflare.com/ACCOUNT_GUID/configurations) under Preferences, change the Account Name from “admin@example.com’s Account” to “ComanyName’s company Account”. Setting this value here will make it easier to work with child accounts when we introduce them later.</p>
</blockquote>

<blockquote>
  <p>This account you are creating is the root of your company’s digital identity as it will control your domain names. You want to ensure that if something catastrophic happens you can recover it via Cloudflare’s account recovery process. Even if you are just using the free tier, on the Billing page (https://dash.cloudflare.com/ACCOUNT_GUID/billing) enter a payment instrument with an accurate billing address.</p>
</blockquote>

<blockquote>
  <p>Once your account email and payment instrument is verified, on the Authentication Management profile page setup “Mobile App Authentication” app based second factor on this root account. Use Cloudflare’s documentation for this step. Once this second factor is setup, capture the recovery codes for this account in your password manager. These codes will allow anyone with access to them to log into the account. If you ever need to “break the glass” and access this account you will need access to these codes. Add these to your company’s disaster recovery documentation.</p>
</blockquote>

<blockquote>
  <h2 id="invite-child-accounts">Invite Child Accounts</h2>
</blockquote>

<blockquote>
  <p>We will now create the child accounts that employees will use for their day-to-day operations. On the Members page (https://dash.cloudflare.com/ACCOUNT_GUID/members) start by enabling the “Member 2FA enforcement” option. Since Cloudflare only supports app and hardware based 2FA, this setting ensures all child accounts that can access our account company account must have strong MFA enabled. A huge shout out to Cloudflare for offering this feature on their free tier, and making it so easy to enable!</p>
</blockquote>

<blockquote>
  <p>Our child accounts will make use of Cloudflare’s Account-scoped roles. These control the permissions the account will have. Since we’re setting up an admin user we’re going to make use of the Super Administrator - All Privileges role. However, as you create other child accounts for users at your company consider what permissions a user needs to complete their tasks. The best way to limit the impact of a breach is for the user to not have the access to steal in the first place. Some useful roles to know about at the Billing and DNS roles. The former only can only edit billing information on the account, great to allow your finance people to manage an account without having any technical access to the account. The latter allows for the management of DNS records. This can be useful for web developers, or IT admins who might need to set these records, but do not need the ability to configure the Cloudflare account itself.</p>
</blockquote>

<blockquote>
  <p>Use the  blue “Invite” button to create a child accounts. Enter the email of the user who’s going to be in charge of your Cloudflare account. Set their “Scope” to include all domains, and select the Super Administrator - All Privileges role. Then click the “Continue to summary” button. Confirm that you have entered the correct info, then click “Invite”</p>
</blockquote>

<blockquote>
  <p>With the invite sent out to child user with administrative privileges, log out of the admin account. Do not use this account again unless you are in a “break glass” scenario. Your child admin account has all the permissions in the parent account to serve as an admin account. If this child account is ever compromised the root account can be used to revoke their access and contain a breach.</p>
</blockquote>

<blockquote>
  <p>Accept the invite with your individual email account. If you don’t already have a Cloudflare account, you will need to create one. When you accept the invite from the parent account, you will be required to add 2FA to the account. When you log into your child account make sure to select the Company Account from the list of accounts. With this structure in place you now have a functional Cloudflare account ready for your staff to operate out of.</p>
</blockquote>

<blockquote>
  <h2 id="benefits-of-a-hardened-account-structure">Benefits of a Hardened Account Structure</h2>
</blockquote>

<blockquote>
  <p>Since your staff will have their own individual accounts, you can keep your root account locked down. The root is the ultimate control of your Cloudflare account, and if you find yourself with a compromised individual account you will be able to revoke their access using the root account. Other than this type of “break glass” scenario this account should never be used. Unfortunately, Cloudflare doesn’t support a native way to be notified if this account is ever used. However, they do have a robust audit trail so you can develop automation to detect any use of this account.</p>
</blockquote>

<blockquote>
  <p>The combination of individual accounts, and Cloudflare’s audit log also allows you to know who is doing what in an account. This principal of non-repudiation is a key security component that you gain when employing this recommended account structure.</p>
</blockquote>

<blockquote>
  <p>Scoped permissions also create boundaries of how large an issue can spread. If a contractor only has the ability to modify the DNS records for a single site, then they can only take down that one site by breaking it’s DNS records.</p>
</blockquote>

<blockquote>
  <p>At Zatik we had to build this runbook for ourselves as we were setting up our DNS infrastructure since we couldn’t find a similar step-by-step, vendor neutral, free guide, so we’re sharing our research in line with our Company Values to pay it forward to the next founder.</p>
</blockquote>]]></content><author><name></name></author><category term="Zatik" /><summary type="html"><![CDATA[Original post on Zatik’s blog]]></summary></entry><entry><title type="html">Secure Domain Registration for Startup Companies</title><link href="z1g1.net/zatik/2024/04/22/secure-domain-registration.html" rel="alternate" type="text/html" title="Secure Domain Registration for Startup Companies" /><published>2024-04-22T00:00:00+00:00</published><updated>2024-04-22T00:00:00+00:00</updated><id>z1g1.net/zatik/2024/04/22/secure-domain-registration</id><content type="html" xml:base="z1g1.net/zatik/2024/04/22/secure-domain-registration.html"><![CDATA[<p>Original post on <a href="https://www.zatik.io/blog/secure-domain-registration">Zatik Blog</a></p>

<blockquote>
  <blockquote>
    <p>As an entrepreneur with an idea, one of the first things you will buy for your business is a domain name. This name is the root of your company’s identity. If you lose control, the company could see its website defaced, customer funds stolen/diverted, suffer downtime on websites and APIs, or see a further compromise of corporate assets as emails are redirected. 
When we started Zatik Security we wanted to register our domain in the most secure way possible. However, we couldn’t find any step-by-step, vendor neutral, free guidance, so we’re sharing our research in line with our Company Values to pay it forward to the next founder.
None of the companies evaluated sponsored Zatik’s assessment or influenced our recommendations. These are our preferences, based on our research of publicly available feature documentation and professional experience as of March 2024. Gentlepersons can disagree, and your experience might be different.</p>
  </blockquote>
</blockquote>

<blockquote>
  <p>As a startup or small business, a full-fledged enterprise domain registry solution probably isn’t necessary or cost effective for your business yet. From a security standpoint, an ideal registrar would support the following six features: 
a table of security features that a domain registrar should have</p>
</blockquote>

<blockquote>
  <p>With these requirements in mind, I reviewed five domain Registrars frequently recommended in startup communities. Since our focus is enabling startups and small businesses, any features locked behind an “enterprise” tier have been excluded. The vendors we reviewed, in alphabetical order, were: Cloudflare, GoDaddy, Hover, Namecheap, and Squarespace. 
A table describing the results of Zatik Securit’s review of domain registrars</p>
</blockquote>

<blockquote>
  <p>Cloudflare meets five of the six criteria that Zatik recommends.</p>
</blockquote>

<blockquote>
  <ol>
    <li>2FA: Requires anyone you invite use Two Factor Authentication (2FA). It is great to see a vendor offer this control on a non-enterprise plan with a simple checkbox.</li>
    <li>Forbid SMS 2FA: Since Cloudflare only supports security key or app based second factors, you can have confidence that staff are not using SMS as second factor.</li>
    <li>RBAC: Invite multiple users to your account with custom user access controls.</li>
    <li>Audit Trail: The audit trail feature is very detailed and includes login information, but customers need to manually script notifications.</li>
    <li>Forced Logout: Sessions can be destroyed using the Revoke Session feature. This allows you to log out an account which may be compromised.</li>
  </ol>
</blockquote>

<blockquote>
  <p>What criteria was not met?  Security Alerts: Cloudflare email notifications do not allow you to set up alerts for other members of your Cloudflare organization. They are only sent to the account’s email itself, so are not a reliable option to monitor for strange logins for other accounts</p>
</blockquote>

<blockquote>
  <p>The full notes of my review can be found in the “Show Your Work” section at the end of this blog post. 
But I already have a domain purchased…</p>
</blockquote>

<blockquote>
  <p>After conducting our review and identifying an ideal registrar, we only had one problem…we had bought zatik.io using Namecheap. Since domain owners are allowed to move their domains between registrars, we started the process of moving our DNS, and domain registration into a hardened Cloudflare account.</p>
</blockquote>

<blockquote>
  <p>The first part of moving a domain name is to ensure the destination registrar supports the Top Level Domain (TLD) of the domain you want to move. Since we’re moving to Cloudflare, we confirmed that .io was one of their supported TLDs. Moving a domain has some restrictions which can prevent transferring registrars. The most common is that a domain cannot transfer if its WHOIS registration information has been modified within 60 days. Cloudflare provides a full list of reasons why a domain could be restricted from transfer into their system; you should review these materials before starting a domain name transfer.</p>
</blockquote>

<blockquote>
  <p>When you plan to move a domain, you must ensure that any existing DNS entries at the source registrar are copied to the destination registrar. If they don’t match, you’ll end up causing an outage for your existing customers/users. I recommend having a second person compare the records at the source and destination registrars after you create them. The DNS records at the destination registrar will not have any impact until you change the nameservers at the source registrar to point to the destination. Once you make this change, DNS records will begin to update. Zatik makes use of several SaaS products for our employees, so I created a checklist before making any changes so I could verify the transfer process was completed.</p>
</blockquote>

<blockquote>
  <p>Once the DNS change was completed, I began the process of transferring the registrar of Zatik.io. This process updates the business relationship to your domain and controls the company you will be renewing the domain with. To begin, you must unlock the domain at the source registrar, which will provide you with a verification code. Enter the code in the destination registrar to complete the transaction. When planning a move, you should budget 10 days into your timeline: your source registrar has up to 5 days to provide you the code, and another 5 to send it to your destination registrar. (It doesn’t usually take 10 days, but it can per the ICANN rules, so it’s good to set the time aside just in case.) The task to move to Cloudflare allowed Zatik to take advantage of their additional account security features. Now you can, too!</p>
</blockquote>]]></content><author><name></name></author><category term="Zatik" /><summary type="html"><![CDATA[Original post on Zatik Blog]]></summary></entry><entry><title type="html">Set your iPhone into Greyscale mode on a schedule</title><link href="z1g1.net/productivity/2023/10/14/greyscale-ios.html" rel="alternate" type="text/html" title="Set your iPhone into Greyscale mode on a schedule" /><published>2023-10-14T00:00:00+00:00</published><updated>2023-10-14T00:00:00+00:00</updated><id>z1g1.net/productivity/2023/10/14/greyscale-ios</id><content type="html" xml:base="z1g1.net/productivity/2023/10/14/greyscale-ios.html"><![CDATA[<p>Your iPhone can be a blackhole of attention. One way to reduce the amount of screentime you spend on it is to make the screen less appealing to look at. You can accomplish this by enabling the Grayscale color filter. This is a <a href="https://support.apple.com/guide/iphone/change-color-and-brightness-iph3e2e1fb0/ios">feature</a> built into iOS to make items easier to see on screen. You can pair this with iOS’s <a href="https://support.apple.com/guide/shortcuts/enable-or-disable-a-personal-automation-apd602971e63/ios">Shortcuts Automation</a> to flip your screen into black and white during your productivity periods.</p>

<p>The Grayscale Color Filter can be flipped on any time by going to <code class="language-plaintext highlighter-rouge">Settings</code> &gt; <code class="language-plaintext highlighter-rouge">Accessibility</code> &gt; <code class="language-plaintext highlighter-rouge">Display &amp; Text Size</code> &gt; <code class="language-plaintext highlighter-rouge">Color Filters</code>&gt; <code class="language-plaintext highlighter-rouge">Grayscale</code>. This will make the screen permenatily in greyscale. There are sometimes when you want this to be enabled and sometimes when you do not, so you can toggle this feature on an off via a triple click of the power button using the <a href="https://support.apple.com/en-us/HT204390">Accessibility Shortcut</a> feature of iOS. Go to <code class="language-plaintext highlighter-rouge">Settings</code> &gt; <code class="language-plaintext highlighter-rouge">Accessibility</code> &gt; <code class="language-plaintext highlighter-rouge">Accessibility Shortcut</code> (1) &gt; <code class="language-plaintext highlighter-rouge">Color Filters</code> (2). This will enable the last Color Filter that you made use of</p>

<p><img src="/assets/images/greyscale-ios-02.png" alt="Enable the accessibility shortcut" /></p>

<p>This is good for one off toggling of this feature, but by pairing it with Shortcuts Automation you can have your phone enable this mode without you having to remember. If you have not installed the <a href="https://apps.apple.com/us/app/shortcuts/id915249334">Shortcuts app</a> you will need to install it from the iOS App Store to follow these instructions. There are 4 Automations that I have setup: Turn on Grayscale at 9:45 AM on Weekdays, Turn Grayscale off at 5 PM on Weekdays, Turn on Grayscale when Bedtime mode is enabled, Turn off Grayscale when Bedtime is disabled. This pattern gives me Grayscale mode when I need to be focusing on other things, or when I should put my phone down around bedtime.</p>

<p>To create the time based Automations, open up the Shortcuts app and select <code class="language-plaintext highlighter-rouge">Automation</code> at the bottom of the screen (this will be between <code class="language-plaintext highlighter-rouge">Shortcuts</code> and <code class="language-plaintext highlighter-rouge">Gallery</code>), then click the plus sign at the top right.</p>

<ol>
  <li>Click on the “Create Personal Automation” button</li>
  <li>Choose the “Time of Day” option</li>
  <li>Enter the time of day you want Grayscale to be enabled/disabled</li>
  <li>Set the Repeat to “Weekly”, and uncheck the circles for Saturday and Sunday.</li>
  <li>Click the “Add Action” button</li>
  <li>Search for “Set color filter”</li>
  <li>Set the value to “On” for your Automation to enable Grayscale, and “Off” for your Automation to turn it off at the end of the day</li>
  <li>On the Review screen ensure you turn off the “Ask Before Running” option to have things run unprompted.</li>
  <li>Tap “Don’t Ask”</li>
  <li>Your screen should look like this. Then click “Done”</li>
</ol>

<p><img src="/assets/images/greyscale-ios-03.png" alt="Step 1 of the process" />
<img src="/assets/images/greyscale-ios-04.png" alt="Step 2 of the process" />
<img src="/assets/images/greyscale-ios-05.png" alt="Step 3 of the process" /></p>

<p>To create the event based Automations follow the same steps as above, but on step 2 select the “Sleep” option vs “Time of Day”.</p>

<ol>
  <li>Click on the “Create Personal Automation” button</li>
  <li>Choose the “Sleep” option</li>
  <li>Select “Wind down begins” for the Automation when Grayscale should be enabled, and “Waking Up” when Grayscale should be disabled.</li>
  <li>Click the “Add Action” button</li>
  <li>Search for “Set color filter”</li>
  <li>Set the value to “On” for your Automation to enable Grayscale, and “Off” for your Automation to turn it off at the end of the day</li>
  <li>On the Review screen ensure you turn off the “Ask Before Running” option to have things run unprompted.</li>
  <li>Tap “Don’t Ask”</li>
  <li>Your screen should look like this. Then click “Done”</li>
</ol>

<p><img src="/assets/images/greyscale-ios-06.png" alt="Step 4 of the process" />
<img src="/assets/images/greyscale-ios-07.png" alt="Step 5 of the process" />
<img src="/assets/images/greyscale-ios-08.png" alt="Step 6 of the process" /></p>

<p>With these four Automations in play your iPhone will enable and disable Grayscale on a schedule</p>]]></content><author><name></name></author><category term="Productivity" /><summary type="html"><![CDATA[Your iPhone can be a blackhole of attention. One way to reduce the amount of screentime you spend on it is to make the screen less appealing to look at. You can accomplish this by enabling the Grayscale color filter. This is a feature built into iOS to make items easier to see on screen. You can pair this with iOS’s Shortcuts Automation to flip your screen into black and white during your productivity periods. The Grayscale Color Filter can be flipped on any time by going to Settings &gt; Accessibility &gt; Display &amp; Text Size &gt; Color Filters&gt; Grayscale. This will make the screen permenatily in greyscale. There are sometimes when you want this to be enabled and sometimes when you do not, so you can toggle this feature on an off via a triple click of the power button using the Accessibility Shortcut feature of iOS. Go to Settings &gt; Accessibility &gt; Accessibility Shortcut (1) &gt; Color Filters (2). This will enable the last Color Filter that you made use of This is good for one off toggling of this feature, but by pairing it with Shortcuts Automation you can have your phone enable this mode without you having to remember. If you have not installed the Shortcuts app you will need to install it from the iOS App Store to follow these instructions. There are 4 Automations that I have setup: Turn on Grayscale at 9:45 AM on Weekdays, Turn Grayscale off at 5 PM on Weekdays, Turn on Grayscale when Bedtime mode is enabled, Turn off Grayscale when Bedtime is disabled. This pattern gives me Grayscale mode when I need to be focusing on other things, or when I should put my phone down around bedtime. To create the time based Automations, open up the Shortcuts app and select Automation at the bottom of the screen (this will be between Shortcuts and Gallery), then click the plus sign at the top right. Click on the “Create Personal Automation” button Choose the “Time of Day” option Enter the time of day you want Grayscale to be enabled/disabled Set the Repeat to “Weekly”, and uncheck the circles for Saturday and Sunday. Click the “Add Action” button Search for “Set color filter” Set the value to “On” for your Automation to enable Grayscale, and “Off” for your Automation to turn it off at the end of the day On the Review screen ensure you turn off the “Ask Before Running” option to have things run unprompted. Tap “Don’t Ask” Your screen should look like this. Then click “Done” To create the event based Automations follow the same steps as above, but on step 2 select the “Sleep” option vs “Time of Day”. Click on the “Create Personal Automation” button Choose the “Sleep” option Select “Wind down begins” for the Automation when Grayscale should be enabled, and “Waking Up” when Grayscale should be disabled. Click the “Add Action” button Search for “Set color filter” Set the value to “On” for your Automation to enable Grayscale, and “Off” for your Automation to turn it off at the end of the day On the Review screen ensure you turn off the “Ask Before Running” option to have things run unprompted. Tap “Don’t Ask” Your screen should look like this. Then click “Done” With these four Automations in play your iPhone will enable and disable Grayscale on a schedule]]></summary></entry><entry><title type="html">Using Jupyter Notebooks in Google Colab and publishing to Google Sheets</title><link href="z1g1.net/jupyter/2023/02/05/Jupyter-Notebook-in-Google-Colab.html" rel="alternate" type="text/html" title="Using Jupyter Notebooks in Google Colab and publishing to Google Sheets" /><published>2023-02-05T00:00:00+00:00</published><updated>2023-02-05T00:00:00+00:00</updated><id>z1g1.net/jupyter/2023/02/05/Jupyter-Notebook-in-Google-Colab</id><content type="html" xml:base="z1g1.net/jupyter/2023/02/05/Jupyter-Notebook-in-Google-Colab.html"><![CDATA[<p><a href="https://jupyter.org/try-jupyter/lab/">Jupyter Lab notebooks</a>  are a great tool for doing data analysis that is either to big for Excel, or you need to repeat the output. <a href="https://colab.research.google.com/">Google Colab</a> is a hosted version of Jupyter Labs. A useful feature of Jupyter Notebooks are that you can share a notebook with someone and they can see the same output. Google Colab adds the extra bonus of allowing you to share your instance of the notebook directly with anyone who has a Google Account, or corporate Google Apps Account. These sharing features also extend to accessing data directly from Google Drive using Google’s <a href="https://github.com/googlecolab/colabtools/tree/main/google/colab">colab library</a>.</p>

<p>If you are collaborating with a user who you don’t want to send to the notebook, you can also make use of colab features to send data directly to Google Sheets. The setups below walk though the process of using the Pandas library to analyze some of Google Colab’s <a href="https://developers.google.com/machine-learning/crash-course/california-housing-data-description">California Housing sample data</a>, and send the results to Google Sheets. Details are also in this <a href="https://github.com/z1g1/notebooks/blob/main/Colab_demo.ipynb">sample notebook</a>.</p>

<h2 id="step-by-step">Step by Step</h2>
<ol>
  <li>Create a new notebook on  <a href="https://colab.research.google.com/">Google Colab</a></li>
  <li>Import both the <code class="language-plaintext highlighter-rouge">pandas</code> and <code class="language-plaintext highlighter-rouge">google.cloab</code> libraries. You can use the <code class="language-plaintext highlighter-rouge">#@title ...</code> nomenclature to title a Python code cell in your Notebook. These will appear in the table of contents allowing you to jump around your notebook.
<img src="/assets/images/google-colab-01.png" alt="python code importing colab and pandas libraries" /></li>
  <li>Use the <code class="language-plaintext highlighter-rouge">colab</code> library access the sample data.  To access the path for an item in Google Drive open the file browser (1), navigate to the file and click the three dot menu (2), then select “copy path” (3) 
<img src="/assets/images/google-colab-02.png" alt="use copy path to get an object's Google Drive path" /></li>
  <li>Create a Pandas Dataframe with the sample data, then use the <code class="language-plaintext highlighter-rouge">df.shape</code> method to see this data has 3000 rows and 9 columns 
<img src="/assets/images/google-colab-03.png" alt="code cell showing accessing the sample data" /></li>
  <li>To create a new Google Sheet for this data you need to authenticate your notebook to your Google Account.  You will use this auth with the <a href="https://docs.gspread.org/en/v5.7.0/index.html">gspread</a> library. 
<img src="/assets/images/google-colab-04.png" alt="code cell showing using gspread authentication" /></li>
  <li>Using <code class="language-plaintext highlighter-rouge">gspread</code> to create a new spreadsheet and worksheet to the size of your data. then using <code class="language-plaintext highlighter-rouge">worksheet.update</code> put the data into Google Sheets
<img src="/assets/images/google-colab-05.png" alt="Code cell using gspread update" /></li>
</ol>

<p>This will have your data into Google Sheets but the one line of <code class="language-plaintext highlighter-rouge">worksheet.update([df.columns.values.tolist()] + df.values.tolist())</code> is doing a lot. How did this data actually get written.</p>

<p>The <a href="https://docs.gspread.org/en/v5.7.0/api/models/worksheet.html#gspread.worksheet.Worksheet.update">worksheet.update</a> method takes in a list of data as the <code class="language-plaintext highlighter-rouge">values</code> attribute. If the elements of this list are themselves lists gspread will send this data as rows and columns to write to the sheet. For example if your data is <code class="language-plaintext highlighter-rouge">data = [[1,1],[2,1],[3,1]]</code> when this is written to Google Sheets it will be written as three rows, with two columns each</p>

<p><img src="/assets/images/google-colab-06.png" alt="a code cell passing a list of lists to gspread" /></p>

<p><img src="/assets/images/google-colab-07.png" alt="the output of the code cell above in Google Sheets" /></p>

<p>With this information we can keep diving into how <code class="language-plaintext highlighter-rouge">worksheet.update([df.columns.values.tolist()] + df.values.tolist())</code> wrote our data to Google Sheets. <code class="language-plaintext highlighter-rouge">df.columns.values.tolist()</code> calls the columns attribute from the Pandas Dataframe. This is then converted to a list using <code class="language-plaintext highlighter-rouge">tolist()</code>. In order to have our list of lists as above this list is included inside of <code class="language-plaintext highlighter-rouge">[ ]</code> inside the <code class="language-plaintext highlighter-rouge">worksheet.update</code>. If this were the only data sent to Google Sheets it would only write the column names</p>

<p>To get each of the remaining 3000 rows of data <code class="language-plaintext highlighter-rouge">df.values.tolist()</code> is used to export the Dataframe’s data to a list. This is then added to the column names list using the <a href="https://docs.python.org/3/reference/datamodel.html#object.__add__">add operator</a>. When this add operation is complete we have a list with 3001 elements, each of which is a row to write to Google Sheets.</p>

<p><img src="/assets/images/google-colab-08.png" alt="code cell showing how the columns and data were put together for Google Sheets" /></p>

<h2 id="whats-next">What’s next</h2>
<p>This example only wrote the Dataframe directly to Google Sheets. However, you can use these same basics to write the output of any of your data analysis  Dataframes or series to Google Sheets. You can also use gspread to update existing worksheets, or create multiple sheets for each of the answers you produce.</p>]]></content><author><name></name></author><category term="Jupyter" /><summary type="html"><![CDATA[Jupyter Lab notebooks are a great tool for doing data analysis that is either to big for Excel, or you need to repeat the output. Google Colab is a hosted version of Jupyter Labs. A useful feature of Jupyter Notebooks are that you can share a notebook with someone and they can see the same output. Google Colab adds the extra bonus of allowing you to share your instance of the notebook directly with anyone who has a Google Account, or corporate Google Apps Account. These sharing features also extend to accessing data directly from Google Drive using Google’s colab library. If you are collaborating with a user who you don’t want to send to the notebook, you can also make use of colab features to send data directly to Google Sheets. The setups below walk though the process of using the Pandas library to analyze some of Google Colab’s California Housing sample data, and send the results to Google Sheets. Details are also in this sample notebook. Step by Step Create a new notebook on Google Colab Import both the pandas and google.cloab libraries. You can use the #@title ... nomenclature to title a Python code cell in your Notebook. These will appear in the table of contents allowing you to jump around your notebook. Use the colab library access the sample data. To access the path for an item in Google Drive open the file browser (1), navigate to the file and click the three dot menu (2), then select “copy path” (3) Create a Pandas Dataframe with the sample data, then use the df.shape method to see this data has 3000 rows and 9 columns To create a new Google Sheet for this data you need to authenticate your notebook to your Google Account. You will use this auth with the gspread library. Using gspread to create a new spreadsheet and worksheet to the size of your data. then using worksheet.update put the data into Google Sheets This will have your data into Google Sheets but the one line of worksheet.update([df.columns.values.tolist()] + df.values.tolist()) is doing a lot. How did this data actually get written. The worksheet.update method takes in a list of data as the values attribute. If the elements of this list are themselves lists gspread will send this data as rows and columns to write to the sheet. For example if your data is data = [[1,1],[2,1],[3,1]] when this is written to Google Sheets it will be written as three rows, with two columns each With this information we can keep diving into how worksheet.update([df.columns.values.tolist()] + df.values.tolist()) wrote our data to Google Sheets. df.columns.values.tolist() calls the columns attribute from the Pandas Dataframe. This is then converted to a list using tolist(). In order to have our list of lists as above this list is included inside of [ ] inside the worksheet.update. If this were the only data sent to Google Sheets it would only write the column names To get each of the remaining 3000 rows of data df.values.tolist() is used to export the Dataframe’s data to a list. This is then added to the column names list using the add operator. When this add operation is complete we have a list with 3001 elements, each of which is a row to write to Google Sheets. What’s next This example only wrote the Dataframe directly to Google Sheets. However, you can use these same basics to write the output of any of your data analysis Dataframes or series to Google Sheets. You can also use gspread to update existing worksheets, or create multiple sheets for each of the answers you produce.]]></summary></entry><entry><title type="html">Setting up DNS, TLS, and a CDN for a Pelican Static Website</title><link href="z1g1.net/pelican/2022/01/28/Setting-up-DNS.html" rel="alternate" type="text/html" title="Setting up DNS, TLS, and a CDN for a Pelican Static Website" /><published>2022-01-28T00:00:00+00:00</published><updated>2022-01-28T00:00:00+00:00</updated><id>z1g1.net/pelican/2022/01/28/Setting-up-DNS</id><content type="html" xml:base="z1g1.net/pelican/2022/01/28/Setting-up-DNS.html"><![CDATA[<p>The first post in this series covered <a href="{filename}2022-01-15-Setting-up-the-site.md">creating a site with Pelican</a> the second covered <a href="{filename}2022-01-22-Hosting-site-via-s3-and-github-actions.md">S3 hosting and Github Actions</a>. The series will conclude with configuration of DNS, TLS, and a CDN. You will need a domain name, AWS Account, and Cloudflare account to complete this step.</p>

<h2 id="create-a-tls-certificate">Create a TLS certificate</h2>

<ol>
  <li>To enable your distribution to serve over TLS you will <a href="https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html">request a public certificate</a> from Amazon Certificate Manager (ACM)
    <ol>
      <li>For the “Fully qualified domain name” enter the domain name that you will eventually want the Pelican generated site to be accessed by</li>
      <li>Select “Use DNS validation”</li>
      <li>Click Create</li>
    </ol>
  </li>
  <li>At this point the certificate will be created but not valid, you need to prove ownership via DNS. Click on the Certificate ID. In the “Domains” section note the “CNAME name” and “CNAME value”.</li>
  <li>In CloudFlare console for your domain <a href="https://support.cloudflare.com/hc/en-us/articles/360019093151-Managing-DNS-records-in-Cloudflare#h_60566325041543261564371">create a CNAME</a> using the values AWS provided above
    <ol>
      <li>The AWS “CNAME name” corresponds to “Name” in CloudFlare console</li>
      <li>The AWS “CNAM value” corresponds to “Content” in CloudFlare console</li>
      <li>Disable the CloudFlare Proxy Toggle</li>
    </ol>
  </li>
  <li>Wait until the certificate’s status is “Issued” in the AWS console.</li>
</ol>

<h2 id="create-a-cloudfront-distribution">Create a CloudFront Distribution</h2>

<ol>
  <li>Following <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-creating-console.html">AWS’ guide</a> configure a Cloudfront Web Distribution
    <ol>
      <li>Select your bucket created as the “Origin domain”</li>
      <li>For “S3 Bucket Access” select “Yes use OAI”, select create a new one, and have AWS update the bucket policy. An Origin Access Identity is another type of identity within AWS which will allow your bucket to be locked down to just access via CloudFront vs being made public.</li>
      <li>In the “Alternate domain name” section enter the domain you wish to use</li>
      <li>In the “Custom SSL certificate” dropdown select the ACM certificate you created above</li>
      <li>In the “Default root object” enter <code class="language-plaintext highlighter-rouge">index.html</code>. <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DefaultRootObject.html">docs</a></li>
      <li>Leave other options at their default and click Create. You will need to wait to move on until the distribution is deployed until you can test</li>
    </ol>
  </li>
  <li>Once the Distribution is no longer showing as “Deploying” you should be able to see your site being served over TLS via the “Distribution domain name”/index.html</li>
</ol>

<h2 id="configuring-dns">Configuring DNS</h2>

<p>With a static site in the S3 bucket configured for static website hosting, and the CloudFront distribution configured via the “Distribution domain name”. Setting up DNS allows visitors to access this via your purchased domain name.</p>

<ol>
  <li>Within your <a href="https://dash.cloudflare.com">CloudFlare dashboard</a>, under the “Website” menu click on the “Add Site” button. Enter your domain name and click “Add site”. Select the free plan and contiune.</li>
  <li>You will be provided with a review of the DNS records currently setup for the domain. If this is a fesh purchase you will likely see DNS records created by your registrar. Click Contiune.</li>
  <li>You will be prompted to update the domain’s existing nameservers and point them to CloudFlare. This will allow CloudFlare to provide answers when browsers ask for information about your domain. These instructions will be unique for your domain registrar of choice.</li>
  <li>Once you have updated your Name Servers click the “Done, check nameservers” button. If this fails after updating at your registrar give it some time and retry.</li>
  <li>Within the CloudFlare “Quick Start Guide” accept the defaults and click “Finish”. Wait until CloudFlare confirms that your domain is congirued</li>
  <li>Within CloudFlare create a CNAME record with a value of <code class="language-plaintext highlighter-rouge">@</code>, and the “Target” as your CloudFront Distribution Domain Name. Ensure you delete the <code class="language-plaintext highlighter-rouge">https://</code> or you will get an error</li>
  <li>Within CloudFlare select the “SSL/TLS” menu and set the encryption mode to “Full”.</li>
</ol>

<p>Your site should now load at the domain name that you requested</p>]]></content><author><name></name></author><category term="Pelican" /><summary type="html"><![CDATA[The first post in this series covered creating a site with Pelican the second covered S3 hosting and Github Actions. The series will conclude with configuration of DNS, TLS, and a CDN. You will need a domain name, AWS Account, and Cloudflare account to complete this step. Create a TLS certificate To enable your distribution to serve over TLS you will request a public certificate from Amazon Certificate Manager (ACM) For the “Fully qualified domain name” enter the domain name that you will eventually want the Pelican generated site to be accessed by Select “Use DNS validation” Click Create At this point the certificate will be created but not valid, you need to prove ownership via DNS. Click on the Certificate ID. In the “Domains” section note the “CNAME name” and “CNAME value”. In CloudFlare console for your domain create a CNAME using the values AWS provided above The AWS “CNAME name” corresponds to “Name” in CloudFlare console The AWS “CNAM value” corresponds to “Content” in CloudFlare console Disable the CloudFlare Proxy Toggle Wait until the certificate’s status is “Issued” in the AWS console. Create a CloudFront Distribution Following AWS’ guide configure a Cloudfront Web Distribution Select your bucket created as the “Origin domain” For “S3 Bucket Access” select “Yes use OAI”, select create a new one, and have AWS update the bucket policy. An Origin Access Identity is another type of identity within AWS which will allow your bucket to be locked down to just access via CloudFront vs being made public. In the “Alternate domain name” section enter the domain you wish to use In the “Custom SSL certificate” dropdown select the ACM certificate you created above In the “Default root object” enter index.html. docs Leave other options at their default and click Create. You will need to wait to move on until the distribution is deployed until you can test Once the Distribution is no longer showing as “Deploying” you should be able to see your site being served over TLS via the “Distribution domain name”/index.html Configuring DNS With a static site in the S3 bucket configured for static website hosting, and the CloudFront distribution configured via the “Distribution domain name”. Setting up DNS allows visitors to access this via your purchased domain name. Within your CloudFlare dashboard, under the “Website” menu click on the “Add Site” button. Enter your domain name and click “Add site”. Select the free plan and contiune. You will be provided with a review of the DNS records currently setup for the domain. If this is a fesh purchase you will likely see DNS records created by your registrar. Click Contiune. You will be prompted to update the domain’s existing nameservers and point them to CloudFlare. This will allow CloudFlare to provide answers when browsers ask for information about your domain. These instructions will be unique for your domain registrar of choice. Once you have updated your Name Servers click the “Done, check nameservers” button. If this fails after updating at your registrar give it some time and retry. Within the CloudFlare “Quick Start Guide” accept the defaults and click “Finish”. Wait until CloudFlare confirms that your domain is congirued Within CloudFlare create a CNAME record with a value of @, and the “Target” as your CloudFront Distribution Domain Name. Ensure you delete the https:// or you will get an error Within CloudFlare select the “SSL/TLS” menu and set the encryption mode to “Full”. Your site should now load at the domain name that you requested]]></summary></entry><entry><title type="html">Hosting a Static Pelican Site in S3 with GitHub Actions</title><link href="z1g1.net/pelican/2022/01/22/Hosting-site-via-s3-and-github-actions.html" rel="alternate" type="text/html" title="Hosting a Static Pelican Site in S3 with GitHub Actions" /><published>2022-01-22T00:00:00+00:00</published><updated>2022-01-22T00:00:00+00:00</updated><id>z1g1.net/pelican/2022/01/22/Hosting-site-via-s3-and-github-actions</id><content type="html" xml:base="z1g1.net/pelican/2022/01/22/Hosting-site-via-s3-and-github-actions.html"><![CDATA[<p>With the <a href="{filename}2022-01-15-Setting-up-the-site.md">static site already created</a> you could manually copy these files to S3 via the AWS console, or via the S3 CLI. However, this adds friction to the writing process. When the files are sent to GitHub via the <code class="language-plaintext highlighter-rouge">git push</code> command they can be automatically be sent to Amazon S3 using <a href="https://docs.github.com/en/actions">GitHub Actions</a> for continuous Deployment/ Continuous Integrations (CI/CD).</p>

<p>When working with any type of system like this ensure you review the <a href="https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions">security</a> and <a href="https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration">pricing</a>. I got a good start using GitHub Actions based on a great post by <a href="https://johnkevinlosito.com/posts/deploy-static-website-to-s3-using-github-actions">johnkevinlosito@</a>.</p>

<h2 id="setting-up-aws-assets">Setting up AWS Assets</h2>

<p>If you do not already have an S3 Bucket <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html">configured for static website hosting</a> complete these steps first. Please note that S3 is a <a href="https://aws.amazon.com/s3/pricing/">paid service</a>. There is a free tier but by using AWS assets you can incur charges.</p>

<ol>
  <li>In the Amazon S3 console locate the ARN of the bucket where you will store the files. It will look something like <code class="language-plaintext highlighter-rouge">arn:aws:s3:::www.z1g1.net</code></li>
  <li>In the <a href="https://console.aws.amazon.com/iamv2/home#/policies">AWS IAM Console</a> Create a new IAM policy which only has the PutObject permission to the specific bucket ARN you captured above. Sample policy below.
    <ol>
      <li><img src="/assets/images/github-actions-z1g1-net-write-only-policy.png" alt="Screenshot of json file with bucket policy" /></li>
      <li>Note that the <code class="language-plaintext highlighter-rouge">Resource</code> entry on line 8 ends with a <code class="language-plaintext highlighter-rouge">*</code>. This is key because if you don’t inclue this your user will not be able to actually overwrite any existing objects</li>
    </ol>
  </li>
  <li>To allow Github Action to make authenticated calls to Amazon S3 an AWS IAM user with limited permissions will be created.
    <ol>
      <li>In the <a href="https://console.aws.amazon.com/iam/home#/users">AWS IAM console</a> create a new user, select the “Access Key - Programatic Access” option.</li>
      <li>Attach the policy that you created above to this user.</li>
      <li>Once you complete this wizard you will be prompted to download credetnaisl for the user. The Access Key ID (AKID) is not sensitive alone. However, the “Secret Access Key” <strong>needs to be kept secret</strong>. Capture this secret for future steps but do not put it in a source code file, or any file which might get uploaded to GitHub. Once you leave this page you will not be able to retreive this information again.</li>
    </ol>
  </li>
</ol>

<p>At this point you will have a bucket which can serve static HTML content. A future hardening step for this infrastrcuture would be to move away from <a href="https://github.com/aws-actions/configure-aws-credentials">IAM users to IAM roles</a>.</p>

<h2 id="configuring-github">Configuring GitHub</h2>

<ol>
  <li>Following the GitHub <a href="https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository">Creating encrypted secrets for a repository</a> documentation store the access key’s secret in the Github Repository for your static site.</li>
  <li>In the root of the static site project create a folder called <code class="language-plaintext highlighter-rouge">.github</code> which will store the .yml file which defines our GitHub Action <code class="language-plaintext highlighter-rouge">mkdir -p .github/workflows</code>. Within this folder create a file called <code class="language-plaintext highlighter-rouge">main.yml</code>
    <ol>
      <li><img src="/assets/images/github-actions-main-yml.png" alt="Screenshot of main.yml file" /></li>
      <li>Lines 3-6 are the trigger for this GitHub Action. Reads as “When there’s a push to the master branch”</li>
      <li>Lines 8-9 starts the deployment job. If there were other actions defined in this file they would be at the same level of indention as line 9</li>
      <li>Line 10 declares what OS will run our deployment action. This files uses the latest Ubuntu image but there are <a href="https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idruns-on">other options</a></li>
      <li>Line 11 starts the steps which will be run on the OS image selected above. <a href="https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions">GitHub action’s purpose</a> is to reduce repettive deployment actions.</li>
      <li>Lines 12 - 13 says to use the second version of the <a href="https://github.com/actions/checkout">checkout action</a>. This action checks-out your repository under $GITHUB_WORKSPACE, so your workflow can access it. If you want to learn what an <code class="language-plaintext highlighter-rouge">action/*</code> does you can look it up under the <code class="language-plaintext highlighter-rouge">https://github.com/ACTION_NAME</code> path.</li>
      <li>Lines 15-20 setup the use of the AWS CLI for use by our worker. These are setup via <a href="https://github.com/aws-actions/configure-aws-credentials">configure-aws-credentials</a></li>
      <li>Line 18 includes the AKID to be used for the upload process. An AKID alone should not be considered a secret. However, this does introduces an Avaliblity issue as this workflow is now tied directly to this AKID. If it’s ever disabled/loses permissions the job will fail.</li>
      <li>Line 19 makes use of the <a href="">GitHub Actions Secrets API</a> to get the value of the <code class="language-plaintext highlighter-rouge">Z1G1_NEW_WRITE_ONLY</code> secret created above. You never want to include the Secret Key for an AWS IAM user within a GitHub repo, or any source code</li>
      <li>Lines 22-23 declare that we will run the <a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html">S3 Sync command</a> to upload the <code class="language-plaintext highlighter-rouge">output</code> directory to the s3 bucket used to host the site. The <code class="language-plaintext highlighter-rouge">--delete</code> at the end of the command will remove files that exist in the destination but not in the source are deleted during sync</li>
    </ol>
  </li>
  <li>When you run a <code class="language-plaintext highlighter-rouge">git push</code> to this repository your output folder will be written to the specified S3 bucket.</li>
</ol>

<h2 id="debugging-deployment">Debugging Deployment</h2>

<p>There are a lot of moving parts to the setup which is created here. Capturing some debugging info if you run into errors.</p>

<ol>
  <li>You need to ensure that your IAM User has permssions to:
    <ol>
      <li>The <code class="language-plaintext highlighter-rouge">PutObject</code> and <code class="language-plaintext highlighter-rouge">ListObject</code> permissions. Both of these are required to run the <a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html">S3 Sync command</a>. In addtion as you want to be able to remove posts from the site, the GitHub action used in this example includes the <code class="language-plaintext highlighter-rouge">--delete</code> option. To acomplish this your IAM user needs to have the <code class="language-plaintext highlighter-rouge">s3:DeleteObject</code> permission as well.</li>
      <li>Has access to both the bucket <code class="language-plaintext highlighter-rouge">"arn:aws:s3:::www.z1g1.net"</code>, and the objects inside it <code class="language-plaintext highlighter-rouge">"arn:aws:s3:::www.z1g1.net/*",</code></li>
    </ol>
  </li>
  <li>You might not be able to run <code class="language-plaintext highlighter-rouge">git push</code> after createing the workflow
    <ol>
      <li>Ensure that your GitHub user / <a href="https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token">personal access token</a> has permissions to create Workflows.</li>
    </ol>
  </li>
  <li>You might get an error saying “actions/checkout@v2 and aws-actions/configure-aws-credentials@v1 are not allowed to be used in …”
    <ol>
      <li>Items listed is Steps of a Github Actions (lines 13, and 16) are code that your runner is executing. It’s convenient to use code that other wrote so you don’t have to rewrite it. However, this does introduce some level of risk as you’re running code others wrote on a system you own. GitHub Actions has 4 levels of permissions to control what can run. These are defined in the repo settings and range (from most to least restrctive): no actions at all, actions owned by the current org/user, actions written by GitHub or orgs they have <a href="https://github.com/marketplace?type=actions&amp;verification=verified_creator">verified</a>, any action.</li>
      <li><img src="/assets/images/github-actions-permissions.png" alt="Screenshot of github actions permissions" /></li>
      <li>My repo is set to use GitHub and their verified plugins. I am trusting GitHub to write files to my S3 bucket so if they wanted to modify them they could, so this is an acceptable level of risk for me.</li>
    </ol>
  </li>
</ol>

<h2 id="example-files">Example files</h2>

<h3 id="sample-iam-policy-for-github-actions">Sample IAM policy for GitHub Actions</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::www.z1g1.net/*",
                "arn:aws:s3:::www.z1g1.net"
            ]
        }
    ]
}
</code></pre></div></div>

<h3 id="sample-mainyml-for-github-actions">Sample main.yml for GitHub Actions</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name: Upload Website via GitHub Actions

on:
  push:
    branches:
    - master

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: AKIAR2JKVZGO665RVFMT
        aws-secret-access-key: $
        aws-region: us-east-1

    - name: Deploy static site to S3 bucket
      run: aws s3 sync ./output/ s3://www.z1g1.net --delete
</code></pre></div></div>]]></content><author><name></name></author><category term="Pelican" /><summary type="html"><![CDATA[With the static site already created you could manually copy these files to S3 via the AWS console, or via the S3 CLI. However, this adds friction to the writing process. When the files are sent to GitHub via the git push command they can be automatically be sent to Amazon S3 using GitHub Actions for continuous Deployment/ Continuous Integrations (CI/CD). When working with any type of system like this ensure you review the security and pricing. I got a good start using GitHub Actions based on a great post by johnkevinlosito@. Setting up AWS Assets If you do not already have an S3 Bucket configured for static website hosting complete these steps first. Please note that S3 is a paid service. There is a free tier but by using AWS assets you can incur charges. In the Amazon S3 console locate the ARN of the bucket where you will store the files. It will look something like arn:aws:s3:::www.z1g1.net In the AWS IAM Console Create a new IAM policy which only has the PutObject permission to the specific bucket ARN you captured above. Sample policy below. Note that the Resource entry on line 8 ends with a *. This is key because if you don’t inclue this your user will not be able to actually overwrite any existing objects To allow Github Action to make authenticated calls to Amazon S3 an AWS IAM user with limited permissions will be created. In the AWS IAM console create a new user, select the “Access Key - Programatic Access” option. Attach the policy that you created above to this user. Once you complete this wizard you will be prompted to download credetnaisl for the user. The Access Key ID (AKID) is not sensitive alone. However, the “Secret Access Key” needs to be kept secret. Capture this secret for future steps but do not put it in a source code file, or any file which might get uploaded to GitHub. Once you leave this page you will not be able to retreive this information again. At this point you will have a bucket which can serve static HTML content. A future hardening step for this infrastrcuture would be to move away from IAM users to IAM roles. Configuring GitHub Following the GitHub Creating encrypted secrets for a repository documentation store the access key’s secret in the Github Repository for your static site. In the root of the static site project create a folder called .github which will store the .yml file which defines our GitHub Action mkdir -p .github/workflows. Within this folder create a file called main.yml Lines 3-6 are the trigger for this GitHub Action. Reads as “When there’s a push to the master branch” Lines 8-9 starts the deployment job. If there were other actions defined in this file they would be at the same level of indention as line 9 Line 10 declares what OS will run our deployment action. This files uses the latest Ubuntu image but there are other options Line 11 starts the steps which will be run on the OS image selected above. GitHub action’s purpose is to reduce repettive deployment actions. Lines 12 - 13 says to use the second version of the checkout action. This action checks-out your repository under $GITHUB_WORKSPACE, so your workflow can access it. If you want to learn what an action/* does you can look it up under the https://github.com/ACTION_NAME path. Lines 15-20 setup the use of the AWS CLI for use by our worker. These are setup via configure-aws-credentials Line 18 includes the AKID to be used for the upload process. An AKID alone should not be considered a secret. However, this does introduces an Avaliblity issue as this workflow is now tied directly to this AKID. If it’s ever disabled/loses permissions the job will fail. Line 19 makes use of the GitHub Actions Secrets API to get the value of the Z1G1_NEW_WRITE_ONLY secret created above. You never want to include the Secret Key for an AWS IAM user within a GitHub repo, or any source code Lines 22-23 declare that we will run the S3 Sync command to upload the output directory to the s3 bucket used to host the site. The --delete at the end of the command will remove files that exist in the destination but not in the source are deleted during sync When you run a git push to this repository your output folder will be written to the specified S3 bucket. Debugging Deployment There are a lot of moving parts to the setup which is created here. Capturing some debugging info if you run into errors. You need to ensure that your IAM User has permssions to: The PutObject and ListObject permissions. Both of these are required to run the S3 Sync command. In addtion as you want to be able to remove posts from the site, the GitHub action used in this example includes the --delete option. To acomplish this your IAM user needs to have the s3:DeleteObject permission as well. Has access to both the bucket "arn:aws:s3:::www.z1g1.net", and the objects inside it "arn:aws:s3:::www.z1g1.net/*", You might not be able to run git push after createing the workflow Ensure that your GitHub user / personal access token has permissions to create Workflows. You might get an error saying “actions/checkout@v2 and aws-actions/configure-aws-credentials@v1 are not allowed to be used in …” Items listed is Steps of a Github Actions (lines 13, and 16) are code that your runner is executing. It’s convenient to use code that other wrote so you don’t have to rewrite it. However, this does introduce some level of risk as you’re running code others wrote on a system you own. GitHub Actions has 4 levels of permissions to control what can run. These are defined in the repo settings and range (from most to least restrctive): no actions at all, actions owned by the current org/user, actions written by GitHub or orgs they have verified, any action. My repo is set to use GitHub and their verified plugins. I am trusting GitHub to write files to my S3 bucket so if they wanted to modify them they could, so this is an acceptable level of risk for me. Example files Sample IAM policy for GitHub Actions { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::www.z1g1.net/*", "arn:aws:s3:::www.z1g1.net" ] } ] } Sample main.yml for GitHub Actions name: Upload Website via GitHub Actions on: push: branches: - master jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 - name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v1 with: aws-access-key-id: AKIAR2JKVZGO665RVFMT aws-secret-access-key: $ aws-region: us-east-1 - name: Deploy static site to S3 bucket run: aws s3 sync ./output/ s3://www.z1g1.net --delete]]></summary></entry><entry><title type="html">Setting up a site with Pelican</title><link href="z1g1.net/pelican/2022/01/15/Setting-up-the-site.html" rel="alternate" type="text/html" title="Setting up a site with Pelican" /><published>2022-01-15T00:00:00+00:00</published><updated>2022-01-15T00:00:00+00:00</updated><id>z1g1.net/pelican/2022/01/15/Setting-up-the-site</id><content type="html" xml:base="z1g1.net/pelican/2022/01/15/Setting-up-the-site.html"><![CDATA[<p>The site here is built using <a href="https://docs.getpelican.com/">Pelican</a> which is a static website generator. It allowed you to write webpages in markdown, then build them along with a config file and theme to HTML files. These files are then stored in an S3 bucket without having to run any webserver. To be able to serve the site via https the site has a certificate created via Amazon Certificate manager and hosted out of CloudFront. To top things off the site is routed over the Cloudflare CDN. I got a good start on this process from a post on Aaron Asaro’s blog about <a href="https://blog.anotherstarburst.com/posts/s3-static-site-cloudflare-ssl/">Serving a static site from S3 using Cloudflare Full SSL</a>. I had to make some tweaks to get SSL full up and working.</p>

<p>You need a command line system to run the commands below. I will be using my <a href="{filename}2022-01-01-setup-dev-system.md">Ubuntu system</a></p>

<h2 id="creating-the-site">Creating the site</h2>
<ol>
  <li>To install <a href="https://docs.getpelican.com/en/latest/quickstart.html">Pelican</a> you will need to have Python and Pip installed.
    <ol>
      <li>On Ubuntu Python 3 is installed by default but you will not be able to use the command <code class="language-plaintext highlighter-rouge">python</code> to run it you would need to use <code class="language-plaintext highlighter-rouge">python3</code>. To save yourself this you can create an alais in your .zshrc file.</li>
      <li><code class="language-plaintext highlighter-rouge">sudo apt install python pip</code></li>
    </ol>
  </li>
  <li>Pelican and all Pyton projects should be installed in a <a href="https://docs.python.org/3/library/venv.html">virtual enviroment (venv)</a>. On Ubuntu, you need to install the python3-ven package via apt
    <ol>
      <li><code class="language-plaintext highlighter-rouge">sudo apt install python3.8-venv</code></li>
    </ol>
  </li>
  <li>create a directory in your projects directory and change to it. Other commands will be run and files executed from here
    <ol>
      <li><code class="language-plaintext highlighter-rouge">mkdir -p ~/projects/SITE_NAME</code> the -p will create the directories in the path</li>
      <li><code class="language-plaintext highlighter-rouge">cd ~/projects/SITE_NAME</code></li>
    </ol>
  </li>
  <li>Once within the site directory you will need to create an activate the virtual environment. The extra step of running inside a venv will allow your system to support work across multiple python projects without dependency issues.
    <ol>
      <li><code class="language-plaintext highlighter-rouge">python3.8 -m venv venv</code> will run the python module venv and create a new environment called venv. You can give it a unique name but it is common to just use the same name. This also allows for command aliasses to be created (below)</li>
      <li><code class="language-plaintext highlighter-rouge">source ./venv/bin/activate</code> this will activate the virtual environment and any modules you install here will be limited to this project.</li>
    </ol>
  </li>
  <li>To make this command easier to work with you can create an alias inside of .zshrc, this is optional
    <ol>
      <li><code class="language-plaintext highlighter-rouge">alias pynv="python3.8 -m venv venv" #New virtual environment</code></li>
      <li><code class="language-plaintext highlighter-rouge">alias pyav="source ./venv/bin/activate" # activate virtual environment</code></li>
      <li><code class="language-plaintext highlighter-rouge">alias pydv="deactivate" # deactivate virtual environment</code></li>
    </ol>
  </li>
  <li>Now you will install Pelican along with the Markdown moduel within the venv with <code class="language-plaintext highlighter-rouge">python -m pip install pelican markdown</code></li>
  <li>Use the <code class="language-plaintext highlighter-rouge">pelican-quickstart</code> commands and walk thru the wizard to fill in some information about your site.
    <ol>
      <li>Enter a value from the “TZ database name” from <a href="https://en.wikipedia.org/wiki/List_of_tz_database_time_zones">this list</a></li>
      <li>I am planning to upload my site to S3 so I will answer yes to that question, and no to the other upload qusetions</li>
    </ol>
  </li>
  <li>There is now the basics of a site which you can see by using the <code class="language-plaintext highlighter-rouge">pelican --listen</code> command. This will show a version of the site that you can access locally at <code class="language-plaintext highlighter-rouge">http://127.0.0.1:8000</code>.
    <ol>
      <li>If you are building the site on a remote host you will need to setup an ssh tunnel to access this site running on local host. I am building on windows so in a <strong>Poweshell</strong> terminal I will run <code class="language-plaintext highlighter-rouge">ssh -L 8080:127.0.0.1:8000 User@Remote_Host -i User_Key_at_Remote_host</code>. This command breaks down as take any trafic on port 8080 for 127.0.0.1 (aka localhost) and send it to port 8000 on the remote machine.</li>
      <li>To get my webbrowser to use this tunnel I am using <a href="https://chrome.google.com/webstore/detail/foxyproxy-standard/gcknhkkoolaabfmlnjonogaaifnjlfnp">FoxyProxy</a> to send all Chrome traffic thru this tunnel. In Proxy Details I set the “Host or IP Address” to localhost and the port to <code class="language-plaintext highlighter-rouge">8080</code>, and checked the <code class="language-plaintext highlighter-rouge">SOCKS proxy</code> box.</li>
    </ol>
  </li>
  <li>While developing your site you might want to run pelican with the auto reload feature. Anytime you make a change to the site’s files the site will be regenerated <code class="language-plaintext highlighter-rouge">pelican --listen -r --ignore-cache</code>. If you are working on a custom theme you can append ````-t path/to/theme``` on the command</li>
  <li>I will be using the default Pelican theme for now, but you can create <a href="https://docs.getpelican.com/en/latest/themes.html">custom themes</a> using this sytem.</li>
  <li>The <code class="language-plaintext highlighter-rouge">pelicanconf.py</code> files contains some values which were set during the <code class="language-plaintext highlighter-rouge">pelican-quickstart</code> process. However, I make some additional changes to further configrue the site
    <ol>
      <li>Remove all the entries from the <code class="language-plaintext highlighter-rouge">LINKS</code> blog roll. This will hide this item from the footer</li>
      <li>Update the <code class="language-plaintext highlighter-rouge">SOCIAL</code> item with links to social providles</li>
      <li>Add a line of <code class="language-plaintext highlighter-rouge">DISPLAY_CATEGORIES_ON_MENU = False</code> this will prevent the site from adding all categories to the menu bar</li>
      <li>Add a line of <code class="language-plaintext highlighter-rouge">DISPLAY_PAGES_ON_MENU = True</code> to add all items created under <code class="language-plaintext highlighter-rouge">content/pages/</code> to the menu bar</li>
      <li>Add a line of <code class="language-plaintext highlighter-rouge">INDEX_SAVE_AS = 'blog_index.html'</code> this moves the default list of items from <code class="language-plaintext highlighter-rouge">content/</code> from the front page of the site to a differnt location at <code class="language-plaintext highlighter-rouge">blog_index.html</code>.</li>
      <li>Use a tupple within <code class="language-plaintext highlighter-rouge">MENUITEMS = (('Posts', 'blog_index.html'),)</code> to add the blog index to the menu bar. You can add additional arbitraty menu items with additional tupples in the <code class="language-plaintext highlighter-rouge">MENUITEMS</code> setting option</li>
      <li>To ensure that the articles have a url structure that maps to post/yyyy/mm/d/title include two of the lines. <code class="language-plaintext highlighter-rouge">43 ARTICLE_URL = 'posts/{date.year}/{date.month}/{date.day}/{slug}.html'</code> and <code class="language-plaintext highlighter-rouge">ARTICLE_SAVE_AS = 'posts/{date.year}/{date.month}/{date.day}/{slug}.html'</code></li>
    </ol>
  </li>
  <li>To create a page to act as the front page create a file in <code class="language-plaintext highlighter-rouge">content/pages/index.md</code>. To ensure it shows as the front page set the following metadata at the top of the page. <code class="language-plaintext highlighter-rouge">URL:</code>, <code class="language-plaintext highlighter-rouge">save_as: index.html</code>, <code class="language-plaintext highlighter-rouge">status: hidden</code></li>
</ol>

<p>At this point you will have the skelton of a site, which as you add content will be written to HTML files in the <code class="language-plaintext highlighter-rouge">output</code> directory when you run the <code class="language-plaintext highlighter-rouge">pelican</code> command. These files can be uploaded to your object store of choice to act as your website. This site is hosted in Amazon S3. The <a href="{filename}2022-01-22-Hosting-site-via-s3-and-github-actions.md">next post in the series</a> covers setting up <a href="https://docs.github.com/en/actions">GitHub Actions</a> for continuous Deployment/ Continuous Intergration (CI/CD).</p>]]></content><author><name></name></author><category term="Pelican" /><summary type="html"><![CDATA[The site here is built using Pelican which is a static website generator. It allowed you to write webpages in markdown, then build them along with a config file and theme to HTML files. These files are then stored in an S3 bucket without having to run any webserver. To be able to serve the site via https the site has a certificate created via Amazon Certificate manager and hosted out of CloudFront. To top things off the site is routed over the Cloudflare CDN. I got a good start on this process from a post on Aaron Asaro’s blog about Serving a static site from S3 using Cloudflare Full SSL. I had to make some tweaks to get SSL full up and working.]]></summary></entry><entry><title type="html">Setting up WSL and Ubuntu on Windows</title><link href="z1g1.net/step%20by%20step/2022/01/01/setup-dev-system.html" rel="alternate" type="text/html" title="Setting up WSL and Ubuntu on Windows" /><published>2022-01-01T00:00:00+00:00</published><updated>2022-01-01T00:00:00+00:00</updated><id>z1g1.net/step%20by%20step/2022/01/01/setup-dev-system</id><content type="html" xml:base="z1g1.net/step%20by%20step/2022/01/01/setup-dev-system.html"><![CDATA[<p>I wanted to get started with writing more online in 2022. I decided to setup a static site hosted out of S3 which I will cover in a future post. However, before taking this step I needed a system that I could work on to build out. I made the check list below for myself and wanted to share it out. As with setting up any computer there are an infinite number of opinions so keep in mind this works for me, but is not the gospel. Where I can I will layout why each step is useful and what it’s doing for context.</p>

<ol>
  <li>I am working on a windows computer so I wanted to start with <a href="https://docs.microsoft.com/en-us/windows/wsl/install">installing WSL2</a> to give myself a Linux command line to operate from. I decided to use <a href="https://wiki.ubuntu.com/WSL#Ubuntu_on_WSL">Ubuntu</a> as the distro to install.</li>
  <li>You can run the distro installed from WSL distro via the start menu. However, the <a href="https://github.com/Microsoft/Terminal">Microsoft Terminal</a> project gives you a number of customization options.</li>
  <li>Under <code class="language-plaintext highlighter-rouge">Startup</code> setup Default profile to WSL distro</li>
  <li>Optional if you want a white background. In the Microsoft Terminal under <code class="language-plaintext highlighter-rouge">Color Schemes</code> create a new Color Scheme with: Foreground Black, Background White, Cursor color Grey, and Selection Background as Grey, Yellow as Gold</li>
  <li>Optional, use <a href="https://github.com/microsoft/terminal/tree/main/src/tools/ColorTool">Color Tool</a> to change the theme. Needs to be extracted and run via <code class="language-plaintext highlighter-rouge">Powershell</code> terminal not Ubuntu</li>
  <li>To use the <a href="https://raw.githubusercontent.com/mbadolato/iTerm2-Color-Schemes/master/schemes/Github.itermcolors">Github</a> theme use <code class="language-plaintext highlighter-rouge">.\ColorTool.exe github.itermcolors</code></li>
  <li>This command and the rest below will take place in an Ubuntu/WSL terminal. To interact at the command line you need to use a shell. The one I use is <a href="https://www.zsh.org/">zsh</a>. There are many options out there, this is the one I use.</li>
  <li>I create a projects directory in my home system as a working directory <code class="language-plaintext highlighter-rouge">mkdir ~/projects</code></li>
  <li>When setting up a linux system you can configure many of the programs on the system with congiguration files stored in your home directoy, colloquially refered to as dotfiles. To make a computer your own these will customize over time, many folks store theirs in a git repo so they don’t loose them. I store mine in a public [GitHub repo])https://github.com/z1g1/Dotfiles). To instal use the Git Clone command within the projects directory <code class="language-plaintext highlighter-rouge">git clone https://github.com/z1g1/z1g1.net.git</code>.</li>
  <li>This will make a copy of all of the dotfiles into the <code class="language-plaintext highlighter-rouge">projects/Dotfiles</code> directory. For programs to find these files and load their config we will use symlinks below</li>
  <li>The benefit of this seperate folder/symlink setup is that it allows you to version control your files without worrying about picking up detritus from your home directory. If you ever make an edit you don’t like you can roll back it via version control.</li>
  <li>Create a symlink off of the Dotfiles repo into the home directory. This will allow for you to source control this file</li>
  <li>.zshrc file <code class="language-plaintext highlighter-rouge">ln -s ~/projects/Dotfiles/.zshrc ~/.zshrc</code></li>
  <li>.tmux.conf file <code class="language-plaintext highlighter-rouge">ln -s ~/projects/Dotfiles/.tmux.conf ~/.tmux.conf</code></li>
  <li>.vimrc file <code class="language-plaintext highlighter-rouge">ln -f -s ~/projects/Dotfiles/.vimrc ~/.vimrc</code></li>
  <li>Set default shell to zsh with <code class="language-plaintext highlighter-rouge">chsh</code> followed by <code class="language-plaintext highlighter-rouge">/usr/bin/zsh</code></li>
</ol>

<p>This setup gives you an Ubuntu system running on windows to act as a foundation for a static webiste</p>]]></content><author><name></name></author><category term="Step by Step" /><summary type="html"><![CDATA[I wanted to get started with writing more online in 2022. I decided to setup a static site hosted out of S3 which I will cover in a future post. However, before taking this step I needed a system that I could work on to build out. I made the check list below for myself and wanted to share it out. As with setting up any computer there are an infinite number of opinions so keep in mind this works for me, but is not the gospel. Where I can I will layout why each step is useful and what it’s doing for context. I am working on a windows computer so I wanted to start with installing WSL2 to give myself a Linux command line to operate from. I decided to use Ubuntu as the distro to install. You can run the distro installed from WSL distro via the start menu. However, the Microsoft Terminal project gives you a number of customization options. Under Startup setup Default profile to WSL distro Optional if you want a white background. In the Microsoft Terminal under Color Schemes create a new Color Scheme with: Foreground Black, Background White, Cursor color Grey, and Selection Background as Grey, Yellow as Gold Optional, use Color Tool to change the theme. Needs to be extracted and run via Powershell terminal not Ubuntu To use the Github theme use .\ColorTool.exe github.itermcolors This command and the rest below will take place in an Ubuntu/WSL terminal. To interact at the command line you need to use a shell. The one I use is zsh. There are many options out there, this is the one I use. I create a projects directory in my home system as a working directory mkdir ~/projects When setting up a linux system you can configure many of the programs on the system with congiguration files stored in your home directoy, colloquially refered to as dotfiles. To make a computer your own these will customize over time, many folks store theirs in a git repo so they don’t loose them. I store mine in a public [GitHub repo])https://github.com/z1g1/Dotfiles). To instal use the Git Clone command within the projects directory git clone https://github.com/z1g1/z1g1.net.git. This will make a copy of all of the dotfiles into the projects/Dotfiles directory. For programs to find these files and load their config we will use symlinks below The benefit of this seperate folder/symlink setup is that it allows you to version control your files without worrying about picking up detritus from your home directory. If you ever make an edit you don’t like you can roll back it via version control. Create a symlink off of the Dotfiles repo into the home directory. This will allow for you to source control this file .zshrc file ln -s ~/projects/Dotfiles/.zshrc ~/.zshrc .tmux.conf file ln -s ~/projects/Dotfiles/.tmux.conf ~/.tmux.conf .vimrc file ln -f -s ~/projects/Dotfiles/.vimrc ~/.vimrc Set default shell to zsh with chsh followed by /usr/bin/zsh This setup gives you an Ubuntu system running on windows to act as a foundation for a static webiste]]></summary></entry></feed>