Jekyll2020-07-12T15:27:57+01:00https://www.andrewhil.es/feed.xmlandrewhil.esPersonal website for Andrew Hiles - Developer from Belfast.Exporting an RDS Postgres database to local database2020-07-05T00:00:00+01:002020-07-05T00:00:00+01:00https://www.andrewhil.es/aws/rds/postgres/2020/07/05/exporting-an-rds-postgres-db-to-local-db<p>If you work with databases in the cloud, there’s a high possibility that you’ve worked with/or at least are familiar with Amazon’s relational database service, <a href="https://aws.amazon.com/rds/">RDS</a>. RDS offers capabilities to host all of the major relational database engines you would need when considering database provisioning, hosting, scaling, etc. In my case, I use RDS almost every day and it’s another AWS service that I think makes developers lives much more hassle-free. My database engine of choice is usually <a href="https://www.postgresql.org/">Postgres</a> and that’s going to be the engine I’ll use for this blog post. However, in most cases, the steps listed below can be easily modified to suit the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html#Welcome.Concepts.DBInstance">other engines</a> RDS supports.</p>
<p>In most cases there a projects where you need to seed a local database with data from a deployed database instance. This could be for testing purposes or debugging some kind of issue in your system. The intention of this blog post is to provide a few brief steps for how this can be achieved when using an RDS Postgres database and <a href="https://www.pgadmin.org/">pgAdmin</a>.</p>
<h1 id="prerequisites">Prerequisites</h1>
<p>I am assuming you have Postgres and pgAdmin installed on your local machine and an RDS Postgres database hosted without any issues on AWS (running on default Postgres 5432 port).</p>
<h1 id="configuring-rds-security-group">Configuring RDS Security Group</h1>
<p>In order to connect to your RDS database instance, you will need to adjust the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithSecurityGroups.html">Security Group</a> configuration on the database to allow for incoming connections from your local machine’s IP address. In most cases, I’d advocate for doing this change via <a href="https://aws.amazon.com/cloudformation/">Cloudformation</a> but for the purpose of this blog post and simplicity, you can alter Security Group configuration via the <a href="https://aws.amazon.com/console/">AWS Console</a>. To modify the Security Group in the AWS Console, locate your RDS database instance, find the Security Group associated to the instance and click ‘Edit inbound rules’. Once inside the rules section, add a new inbound rule with type ‘PostgresSQL’, set protocol to ‘TCP’ with a port range of 5432. Select the Source option of ‘My IP’ and the AWS Console should automatically add your IP address to the rule. Now that you’ve added this rule, you should be able to connect from your local machine directly to the RDS database instance.</p>
<h1 id="connecting-using-pgadmin">Connecting using pgAdmin</h1>
<p>Now that you have configured the Security Group ingress rules to allow connections from your local machine, you need to connect to the database. Using <a href="https://www.pgadmin.org/">pgAdmin</a>, create a new server with the database host URL and user credentials used to access the database.</p>
<h1 id="using-pg_dump-module-to-create-backup">Using pg_dump module to create backup</h1>
<p>Once you have successfully connected to the database, locate the database in the pgAdmin browser panel then right-click on the database name. Select the ‘Backup’ option. This will let you configure the backup task, selecting the dump file location, compression options, etc. Once you are happy with the configuration of the backup, click ‘Backup’. pgAdmin will then run a <a href="https://www.postgresql.org/docs/12/app-pgdump.html">pg_dump</a> command that should look something like the code below.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pg_dump <span class="nt">--file</span> <span class="s2">"/pathToYourDumpFile.sql"</span> <span class="nt">--host</span> <span class="s2">"yourRDSInstance"</span> <span class="nt">--port</span> <span class="s2">"5432"</span> <span class="nt">--username</span> <span class="s2">"username"</span>
</code></pre></div></div>
<h1 id="loading-backup-into-local-database">Loading backup into local database</h1>
<p>Once you have successfully created your .sql dump file, create an empty local database using pgAdmin. Once created, select your local database instance in pgAdmin and right-click ‘Restore’, locate your dump file and click ‘Restore’ to run the job. Depending on how large the file is, this could take some time. Once the job completes, you will have a replica of an RDS database setup for local usage.</p>
<p>It’s worth noting, this would be much easier to explain using commands via the terminal but it also requires setting up user configuration for when you want to connect to a Postgres database and this can be tricky. Using pgAdmin makes this aspect of creating a backup and restoring easier for the purposes of a blog post.</p>
<p>Hopefully this will be of use to someone else. Thanks.</p>If you work with databases in the cloud, there’s a high possibility that you’ve worked with/or at least are familiar with Amazon’s relational database service, RDS. RDS offers capabilities to host all of the major relational database engines you would need when considering database provisioning, hosting, scaling, etc. In my case, I use RDS almost every day and it’s another AWS service that I think makes developers lives much more hassle-free. My database engine of choice is usually Postgres and that’s going to be the engine I’ll use for this blog post. However, in most cases, the steps listed below can be easily modified to suit the other engines RDS supports.Running NPM script tasks in different directories2020-06-27T00:00:00+01:002020-06-27T00:00:00+01:00https://www.andrewhil.es/npm/2020/06/27/running-npm-script-tasks-in-a-different-directory<p>Recently I’d been working on a TypeScript project that had nested modules throughout the codebase. It had a top-level package.json file and three nested sub-modules all with their own package files. In order to build and test the project before checking-in code changes, I needed to access the NPM script tasks of each sub-module to build them and run the unit tests.</p>
<p>Originally, I began writing a bash script that could be executed from a script task in the top-level package.json that would navigate into each sub-module directory, build the code and run the tests. This felt acceptable but after taking a closer look in the <a href="https://docs.npmjs.com/cli-documentation/cli">NPM CLI docs</a>, I came across the awesome <a href="https://docs.npmjs.com/cli/prefix.html">–prefix</a> script option.</p>
<p>Instead of running my bash script, I was able to use –prefix to execute the build and test NPM script tasks of each sub-module all from within the parent package.json file.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build <span class="nt">--prefix</span> ./yourSubModuleDirectory
</code></pre></div></div>
<p>What this does is search for a package.json file in the yourSubModuleDirectory folder and runs the build script task defined in that package file. Pretty awesome, right? Maybe this will come in handy for you some day, too.</p>Recently I’d been working on a TypeScript project that had nested modules throughout the codebase. It had a top-level package.json file and three nested sub-modules all with their own package files. In order to build and test the project before checking-in code changes, I needed to access the NPM script tasks of each sub-module to build them and run the unit tests.Software Engineering - An Industry of Opportunity2020-03-09T00:00:00+00:002020-03-09T00:00:00+00:00https://www.andrewhil.es/general/2020/03/09/software-engineering-an-industry-of-opportunity<p><img src="../../../../images/2020-03-08.png" alt="Software Engineering - An Industry of Opportunity blog post - banner image showing the 5 facts stated by tech folks on twitter but instead shows this blog title instead of real values." title="Software Engineering - An Industry of Opportunity blog post - banner image showing the 5 facts stated by tech folks on twitter but instead shows this blog title instead of real values." /></p>
<p>Last month, lots of people working within the tech industry (Software Engineering, specifically) contributed to a trend on Twitter where they shared their personal response to 5 important facts related to their professional journey (1. Education, 2. Years in the game, 3. Job Title, 4. Working Location, 5. Salary). Personally, I thought this was a really positive endeavour as it set realistic expectations on a ton of different things that working in the tech industry as an Software engineer can bring.</p>
<p><img src="../../../../images/2020-03-08-img1.png" alt="Screenshot of Dan Abramov and Sara Vieira's career facts tweets" title="Screenshot of Dan Abramov and Sara Vieira's career facts tweets" /></p>
<p>Providing such transparency like this helps break down barriers that have been built-up for years and I really believe that folks working as engineers in the industry right now are in such a prosperous position. Over the next few paragraphs, I’ll give my take on how I see things. It’s worth noting, I’m going to tailor my responses based on Software Engineering specifically, as that was what 90% of the tweets stated as their job titles. I did see a few tweets from project managers and scrum masters, etc, but I’m really not qualified to speak from that perspective. With that being said, here’s my two cents.</p>
<h1 id="-education">🏫 Education</h1>
<p>In most cases there used to be a requirement that for entry-level Software Engineering positions, a college degree was required. It didn’t necessarily need to be Computer Science, but a college degree was listed as a requirement in most advertised vacancies. It was almost considered a necessity to get your foot in the door. Personally, this suited me for a long time. Being the proud owner of a Computing degree, it felt right that I had the correct key required to unlock a career in the industry I’d studied for many years to join. However, in recent years, I’ve backtracked on that thought.</p>
<p>Attaining a college degree is a huge achievement. By completing a degree, you are proving that you can study and keep focussed to see something through - the combination of that commitment with an aptitude to learning is nothing to be disregarded. However, getting a degree doesn’t necessarily make someone a better Software engineer, compared to another aspiring engineer without one. Going to university comes at a cost - in most cases, students take on large debts to pay tuition fees and sustain a life whilst studying. Sometimes this can be offputting. In other cases, people might have other dependencies that don’t give them the chance to go to university despite being more than qualified. Even more, some people just don’t want to go into higher education - and that’s okay, too. College isn’t for everybody.</p>
<p>The biggest thing I want to make clear here is that the tide is changing. The need for college degrees isn’t as prominent as it once was and I think that’s a good thing. The barrier to entry for tech jobs is being removed via internships, code academies and a promotion of self-learning. By doing so, the pool of opportunity opens for many folks. I’ve read through countless tweets of very influencial people working in the tech industry that were listing their education as self-taught. In some cases, they might be self-taught and had a college degree in something else not related to Computer Science, etc, but in other cases, they didn’t have any degree at all. If it works for folks at large Software companies like Facebook and Twitter, it can work for anyone else.</p>
<p>Again, it’s worth re-iterating, I still value education and what a degree means, but I strongly believe those without qualifications shouldn’t be denied entry to a career in Software Engineering because they don’t have a graduation day photo, or plaque on the wall. This industry is changing and we’re seeing more people move into amazing careers, with degree or without. I’m all for this. Had I the chance to redo things, I’d have probably avoided going to college and really tried to self-learn, building applications myself with the intention of getting into a Software company via an internship or something that would let me work my way up in the company instead of leaving college in thousands of pounds worth of debt.</p>
<h1 id="-years-in-the-game">⏳ Years in the game</h1>
<p>The tech industry can be a career for life. In the United Kingdom, the average person is expected to change their career 5 times in their lifetime. I can’t see this happening in Software unless folks deliberately seek a change. Software Engineering encapsulates so many different domains - Web Development, Mobile Development, Data Science, Security, Cloud Computing, Automated Intelligence, etc. There are so many areas to work in and too many for even the smartest, most productive folks to master.</p>
<p>From my own experience, I started with an interest in Web programming and quite quickly moved into Cloud Computing, but I know in the next few years my focus will shift to another medium within Software Engineering and that’s nothing but a good thing to me. If you’re passionate about working in technology, I think there’s a career for life on offer. Having the opportunities presented to folks in tech right now is an incredible prospect. Sadly, we’re seeing more and more traditional trades and careers evaporate due to modern-day enhancements, but it’s guaranteed for a long time yet that tech jobs, especially Software jobs, will be needed. This puts anyone currently working, or aspiring to, work in Software, in a fantastic position.</p>
<h1 id="-job-title">🏷 Job Title</h1>
<p>Within Software there are so many different positions available. Too many to list here. This is nothing but a good thing. It shows the levels of opportunity and space to grow that are out there. I think it’s highly motivating to work in an industry where you know you have opportunities to progress. Not every career has this extravagance. In many careers you could work for years on end without a promotion and feel very much like evolution is lacking. Software Engineering is different. Take for example a new hire at an entry level in a company. They normally join with a title of Associate or Junior Engineer and can see in front of them a career path up to Senior, Principal, Staff Engineer, Developer Advocate and above. This isn’t always the case, but in most larger companies, you can change your job title in a good cadence of time throughout your service there. Providing you put the work in, of course.</p>
<h1 id="working-location">🌎Working Location</h1>
<p>The tech industry is thriving right now and has been for some time. There are Software jobs on offer all across the world. However, a large number of technical professionals are not constrained to work in the country where their employer is based. A large number of companies offer remote working opportunities which means a lot of employees in tech can work from the comforts of their own homes, shared workspaces, etc, to do their job. I’ve read different threads on Twitter where folks have shared their experiences travelling the world whilst working remotely, which just proves if you have a computer, appropriate Software and a working internet connection - you can be just as productive in a different place each day as you would be seated at the same desk day in, day out, in an office of permanence. Again, not many careers would be able to support this type of flexibility. We are beyond fortunate.</p>
<h1 id="-salary">💸 Salary</h1>
<p>Probably the biggest thing most people will consider when working in any industry is how much many they get paid. I don’t think Software Engineering is any different. Salaries in the industry are at an all-time high right now. With more and more companies trying their best to recruit the best possible candidates, it creates a competitive environment, which results in lots of companies paying high salaries for all sorts of positions across the Engineering spectrum.</p>
<p>One of the biggest takeaways I took from the sharing of these details on Twitter was different people’s reactions to when <a href="https://twitter.com/dan_abramov">Dan Abramov</a> of React notoriety, <a href="https://twitter.com/dan_abramov/status/1228454264915271683?s=20">shared his facts and figures</a>. He listed that his current salary was £100,000 per year and it seemed to spark widespread shock. I think the general perception was that someone working so prominently in the industry, being so visible and vocal, would’ve been paid a higher amount and plenty of folks made it clear that the consistency between different Engineering salaries was widely skewed. In Dan’s case, he was very transparent in explaining how he earned a generous salary for a Software engineer working in London (he does earn stock on top of his salary, too) and how it would take him to move across the world to most likely capatalize on a larger figure, if he really wanted it. This is another thing to really consider. If you are wanting to make a lot of money, it really depends on your location and negotiation skills. The opportunities are there if you really want them and are prepared to relocate, etc.</p>
<p>From my own experience, working as a Software engineer in Belfast, Northern Ireland, I can affirm that the industry pays well. I work in a city that’s going through a bit of tech boom right now, with lots of different companies setting up outfits looking for Software professionals. Salary figures certainly do vary, but on the whole, they are reasonably well-paying and it doesn’t require relocating to London or San Francisco. Credit to Dan for his honesty as he followed-up his tweet, explaining how he once struggled knowing others in the industry were being paid more than him for similar work, but over time it was something he was able to move past and let go. For me personally, I feel like that was something I once struggled with, too. Thankfully, I managed to let that go a long time ago. In recent times, similar to Dan, I’ve realised that this industry doesn’t really have rules when it comes to salary figures, but I can accept that most folks working in tech have the opportunity to earn a good living for themselves and their loved ones through their careers. If you are money-motivated, you will most likely earn a lot…and let me be clear, there is nothing wrong in it, if that’s your motivation. However, if you are motivated by learning and working hard also, you will find the money will follow, too.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>Looking back, I hope I’ve relayed my take on things in a coherent message and this doesn’t come across as complete rambling. I believe firmly that what folks did on Twitter last month was a great thing. It helped stimulate lots of great conversations on Twitter, in the workplace, at home, etc, about what a career in Software can offer. I have many friends and family members who work in other industries and I’m amazed at what they do, and strangely most of them when discussing work are always curious about what my job entails. Sometimes I think we can take what we do as Software engineers for granted - we mightn’t always feel like the work we’re doing is making a difference, but when we take a step back, it can really help us take stock of what we do; solving problems with the intention of making people’s lives easier.</p>
<p>This industry has a long way to go. It’s not perfect. There are certainly many areas that need improving, but what we do have are a lot of positives to be thankful for. Like being able to get a job with/without college education, work in a thriving industry, wear many different hats and job titles, work from anywhere and sustain a good working income whilst doing all of the above. I welcome more transparency like this in the future and hope it can inspire more and more folks to jump at the opportunity of a career in Software Engineering.</p>
<p>Thanks for reading.</p>
<p>Andrew</p>Last month, lots of people working within the tech industry (Software Engineering, specifically) contributed to a trend on Twitter where they shared their personal response to 5 important facts related to their professional journey (1. Education, 2. Years in the game, 3. Job Title, 4. Working Location, 5. Salary). Personally, I thought this was a really positive endeavour as it set realistic expectations on a ton of different things that working in the tech industry as an Software engineer can bring.Fetching zip files from S3 Buckets with NodeJS2020-03-03T00:00:00+00:002020-03-03T00:00:00+00:00https://www.andrewhil.es/nodejs/aws/serverless/2020/03/03/nodejs-lambda-fetching-s3-zip-files<p>Recently, I’ve been working on a project that uses a lot of Serverless technology. In most cases, I’m writing Serverless functions in AWS using their FaaS (functions as a service) offering, <a href="https://aws.amazon.com/lambda/">AWS Lambda</a>. Amazon have did a great job in providing a <a href="https://aws.amazon.com/sdk-for-node-js/">SDK for NodeJS</a> which makes writing Lambda functions in JavaScript very simple as it provides lots of different APIs for interacting with other AWS services.</p>
<p>Just last week, I was tasked with writing a Lambda function that would receive an SNS event each time a new zip file was uploaded to any S3 Bucket in a specific AWS account. The Lambda had to parse the event metadata to get the file key, fetch the file, unzip the file and then read the data into a temporary file that could be used for further processing. This sounds reasonably straight forward but it took me a moment to re-familiarize myself with NodeJS Streams and how to use them.</p>
<p>I’ve been making a conscious effort to use <a href="https://javascript.info/async-await">async/await</a> in most places where I’m dealing with asynchronous code and the AWS SDK for NodeJS works well with Promises (<a href="https://aws.amazon.com/blogs/developer/support-for-promises-in-the-sdk/">despite them not mentioning this very publicly</a>) so it made sense to take advantage here. The code snippet below shows how I implemented a solution to fetch a zip file from S3, unzip it and write the contents of the file out to a temporary file in the Lambda’s temporary working directory.</p>
<p>PS. It’s worth noting, normally when making a call to get a file from S3, you should be able to access the raw file data by accessing the body of the result. The reason why this task was reasonably cumbersome was because I had to deal with .zip files and streams. Thankfully, using <a href="https://node.readthedocs.io/en/latest/api/zlib/">zlib</a> made this all a bit more manageable.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">aws</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">aws-sdk</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">fs</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">zlib</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">zlib</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">s3</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">S3</span><span class="p">();</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">handler</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// parse event data to get bucket and file details</span>
<span class="kd">const</span> <span class="nx">s3BucketName</span> <span class="o">=</span> <span class="nx">snsMessage</span><span class="p">.</span><span class="nx">Records</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">s3</span><span class="p">.</span><span class="nx">bucket</span><span class="p">.</span><span class="nx">name</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">s3FileKey</span> <span class="o">=</span> <span class="nx">snsMessage</span><span class="p">.</span><span class="nx">Records</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">s3</span><span class="p">.</span><span class="nx">object</span><span class="p">.</span><span class="nx">key</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">s3BucketParams</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">Bucket</span><span class="p">:</span> <span class="nx">s3BucketName</span><span class="p">,</span>
<span class="na">Key</span><span class="p">:</span> <span class="nx">s3FileKey</span>
<span class="p">};</span>
<span class="kd">const</span> <span class="nx">s3ReadStream</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">s3</span><span class="p">.</span><span class="nx">getObject</span><span class="p">(</span><span class="nx">s3BucketParams</span><span class="p">).</span><span class="nx">createReadStream</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">streamS3FileToLocalFile</span> <span class="o">=</span> <span class="p">(</span><span class="nx">inputStream</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nb">Promise</span><span class="p">((</span><span class="nx">resolve</span><span class="p">,</span> <span class="nx">reject</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">fileWriteStream</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">createWriteStream</span><span class="p">(</span><span class="dl">'</span><span class="s1">/tmp/data.txt</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">unzip</span> <span class="o">=</span> <span class="nx">zlib</span><span class="p">.</span><span class="nx">createGunzip</span><span class="p">();</span>
<span class="nx">inputStream</span>
<span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">unzip</span><span class="p">)</span>
<span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">fileWriteStream</span><span class="p">)</span>
<span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="dl">'</span><span class="s1">finish</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">resolve</span><span class="p">({</span> <span class="na">message</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Complete!</span><span class="dl">'</span> <span class="p">});</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="dl">'</span><span class="s1">error</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">reject</span><span class="p">({</span> <span class="na">message</span><span class="p">:</span> <span class="s2">`Something went wrong </span><span class="p">${</span><span class="nx">err</span><span class="p">.</span><span class="nx">message</span><span class="p">}</span><span class="s2">`</span><span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">};</span>
<span class="k">await</span> <span class="nx">streamS3FileToLocalFile</span><span class="p">(</span><span class="nx">s3ReadStream</span><span class="p">);</span>
<span class="c1">// do stuff with file data</span>
<span class="c1">// end lambda execution</span>
<span class="k">await</span> <span class="p">{</span> <span class="na">message</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Successfully fetched S3 file data :)</span><span class="dl">"</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The above code was the solution I ended-up settling with. When you fetch a file from a S3 Bucket, you need to create a read stream. Once you have setup your stream, that is what you pass to the ‘streamS3FileToLocalFile’ function which wraps all of the downstream async function calls in a Promise. If either the unzip or writing to local file steps should encounter an error, the error event emitter will fire and the promise rejected. If the Promise resolves successfully, the file has been fetched from S3, unzipped and it’s contents written to a local file.</p>
<p>This works really well for myself and has been reasonably performant. I was unsure how well the code would handle large zip files containing thousands of lines but it seems like a robust solution. The reason for sharing this was because I struggled to find any good examples on the internet myself so I figured it would be worth sharing once I’d implemented a satisfactory solution that could hopefully help others, and possibly myself, should I ever need to do this again.</p>
<p>Thanks for reading! :)</p>
<p>Andrew</p>Recently, I’ve been working on a project that uses a lot of Serverless technology. In most cases, I’m writing Serverless functions in AWS using their FaaS (functions as a service) offering, AWS Lambda. Amazon have did a great job in providing a SDK for NodeJS which makes writing Lambda functions in JavaScript very simple as it provides lots of different APIs for interacting with other AWS services.Killing Running Processes with NodeJS & NPM2019-08-21T00:00:00+01:002019-08-21T00:00:00+01:00https://www.andrewhil.es/snippets/2019/08/21/killing-running-processes-with-nodejs-and-npm<p>Recently I was trying to use bash/terminal to kill a running process on my local machine. I’d been running a Python service that was hanging and wouldn’t stop so I knew it was going to take something a little more to free-up the port it was hogging. Like with most terminal commands, my memory drew a blank and I had to do a quick Google search to find the exact command I was looking for. There’s a few ways of doing this but here’s the usual way of killing a running process.</p>
<h1 id="using-bashterminal">Using Bash/Terminal</h1>
<p>Get the process attached to the port you need to free-up (e.g. port 4000)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lsof -i :4000
</code></pre></div></div>
<p>Kill the process using the process ID retrieved from the above command</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kill -9 <process ID>
</code></pre></div></div>
<p>The above will kill a process immediately and should make the desired port available.</p>
<h1 id="using-nodejs">Using NodeJS</h1>
<p>After doing a bit more searching, I came across a new way of killing processes that I’ve since started using. There is one prerequisite - it requires <a href="https://nodejs.org/en/">NodeJS</a> and <a href="https://www.npmjs.com/">NPM</a> to be installed. If you have NodeJS and NPM installed, you can run the following command and it will kill any running process on the port specified.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npx kill-port 3000
</code></pre></div></div>
<p>Installing NodeJS and NPM purely for killing running processes seems a bit excessive, but if you already have them installed, this feels like a great alternative compared to the original commands I’ve listed above using bash/terminal. It’s much more concise and reduces the need to lookup process IDs before you can kill a running process.</p>
<p>-Andrew</p>Recently I was trying to use bash/terminal to kill a running process on my local machine. I’d been running a Python service that was hanging and wouldn’t stop so I knew it was going to take something a little more to free-up the port it was hogging. Like with most terminal commands, my memory drew a blank and I had to do a quick Google search to find the exact command I was looking for. There’s a few ways of doing this but here’s the usual way of killing a running process.Winning an AWS IoT Button2019-06-22T00:00:00+01:002019-06-22T00:00:00+01:00https://www.andrewhil.es/iot/aws/2019/06/22/winning-an-aws-iot-button<p>I recently won an AWS IoT Button through a competition in my work. I’m not 100% sure what I’ll do with it as yet but it seems like a pretty cool device and a really intuitive way for anyone to get involved with the internet of things. It shouldn’t come as a surprise that due to it being a Amazon product, it’s tied to AWS (Amazon Web Services) so you’ll have to create an account with them and use their services to handle the programming and integrations that the button will use. I’ve used AWS for approximately 3 years now and I love the capabilities that their services provide to developers, so I’m very excited to get using the IoT Button.</p>
<p><img src="../../../../../images/2019-06-22-img1.jpg" alt="alt text" title="My AWS IoT Button" /></p>
<p>One sad drawback I’ve discovered is the battery cannot be replaced. It’s been soldered inside which would make fitting replacements reasonably difficult to a hardware novice like myself. However, Amazon predict you can get approximately one thousand pushes (and maybe a few more) of the button before the internal battery runs it’s course. So I’m currently trying to think of a few good use cases for how I could make use of it.</p>
<h1 id="burritos-on-demand">Burritos on demand</h1>
<p>Since I work in Belfast and love burritos from <a href="https://www.boojummex.com">Boojum</a>, my current idea would be to implement an integration that when I push the button, a request is sent to Deliveroo for a burrito from Boojum’s central restaurant with the delivery address set to my office. I’m not sure how feasible this would be but it’s probably going to be something I’ll try and implement in the not so distant future.</p>
<p>I’ll be sure to write a follow-up post in the coming weeks to explain what idea I decided on and if my burritos on demand concept ever developed into anything more than a gluttonous dream.</p>
<p>-Andrew</p>I recently won an AWS IoT Button through a competition in my work. I’m not 100% sure what I’ll do with it as yet but it seems like a pretty cool device and a really intuitive way for anyone to get involved with the internet of things. It shouldn’t come as a surprise that due to it being a Amazon product, it’s tied to AWS (Amazon Web Services) so you’ll have to create an account with them and use their services to handle the programming and integrations that the button will use. I’ve used AWS for approximately 3 years now and I love the capabilities that their services provide to developers, so I’m very excited to get using the IoT Button.Facing fears2019-06-19T00:00:00+01:002019-06-19T00:00:00+01:00https://www.andrewhil.es/personal/2019/06/19/facing-fears<p>After writing a <a href="/personal/2018/12/08/done-is-better-than-perfect.html">post</a> 6 months ago talking about how I wanted to blog once per week, I’ve realised that setting such a goal was ridiculous. Life gets busy and sometimes other priorities take precedent. Either way, I’m going to try and make a conscious effort to be more active in my writing here.</p>
<p>However, that’s not to say I haven’t been actively writing elsewhere. Just recently, I wrote a piece for my employer’s internal blogging platform detailing my recent experiences of reviving a legacy software application and deploying it to Production within a 4 month timeframe. The article was titled ‘Resurrecting the past for a modernised future’ and it was met with a considerable amount of positive feedback. So much so, after I published it, multiple colleagues approached me and suggested that converting the article into a talk would be beneficial for others in the local tech scene to hear.</p>
<p>At the time, I wasn’t sure I had the capabilities of being someone who could speak in front of a large crowd of people. My experiences of public speaking in the past have been limited, but the memories I have of those occasions weren’t great. In one occurrence, I can remember completely freezing and a friend having to jump in and save me. Another memory that still lives on was when I got asked a question in a public forum that I couldn’t answer correctly. Revisiting such experiences reminded me that my brain’s memory vault hadn’t discarded those difficult experiences from the past. I started to think that maybe leaving my article to remain purely in textual form on my employer’s blog platform was the best thing.</p>
<h1 id="testing-the-waters">Testing the waters</h1>
<p>Several weeks passed and I kept thinking that I needed to face my fears of public speaking. However, anxiety and imposter syndrome can be quite debilitating for me and each time I began thinking more positively it felt like something would drop from the sky, hit me in the face and remind me that it wasn’t going to happen. Nonetheless, one day I seemed to have much more resilience to the negativity and felt I was ready to do it. Without any hesitation, I performed a swift Google search of local meet-ups in the city of Belfast that I might be able to approach and pitch my idea of a talk to.</p>
<p><img src="../../../../images/2019-06-19-img3.jpg" alt="alt text" title="Northern Ireland Developer Conference logo" /></p>
<p>One of the first results I came across was for a local tech conference, <a href="https://www.nidevconf.com/">NIDevConf(Northern Ireland Developer Conference)</a> and the moment I noticed they had a request for speaker proposals, I started creating a lightning talk proposal for what I wanted to discuss. Lightning talks were limited to 10 minutes in duration and if requested, the speaker could take questions afterwards - this felt like the perfect opportunity for me. Within approximately 30 minutes, I had submitted the proposal and that was it. I’d made the first move and it was up to the organisers of the event to decide whether my talk was worthy of their approval.</p>
<h1 id="acceptance">Acceptance</h1>
<p>Weeks passed by and I hadn’t heard anything about the status of my talk proposal. Then out of the blue I received an email to confirm it had been accepted. Instantly, I was excited at the prospect of facing my fears of public speaking and at the same time, help educate others in how legacy software can be modernised, but I also felt an equal amount of apprehension in taking part.</p>
<p>Eventually, the dust settled and I got over the prospect of standing up in-front of lots of strangers discussing topics that I’m sure lots of them knew more about. I spent several evenings stripping away the fat from my blog article that I’d written, putting what remained into a summarised slide deck. I showed several trusted colleagues my slides to attain feedback, expecting high levels of critique but was surprised to find that they were all very impressed with the content I’d cobbled together. The only significant takeaway was to add a bit of flare to the slides (icons, images, gifs, etc) - which was great advice as it helped me enhance the look and feel of the message I wanted to present.</p>
<p>Once I was happy with the content, it was up to me to make sure I delivered it properly.</p>
<p><img src="../../../../images/2019-06-19-img1.jpg" alt="alt text" title="Twitter message about my talk at Northern Ireland Developer Conference 2019" /></p>
<h1 id="practice-practicepractice-some-more">Practice, practice…practice some more</h1>
<p>In the lead up to my talk, I spent several evenings practising my talk. My girlfriend, mother and cat were probably sick to the back teeth of hearing the same ten minutes of technical jargon and acronyms being mentioned, all whilst pretending to be super interested when really they were only doing it to support me.</p>
<p>Thankfully, after spending time practicing and making small tweaks/amendments to the slide content, I was able to keep the talk to a ten minute duration without needing notes. This was a great help for me as it really meant I could speak freely to my slides, without having to rely on additional aids and made my talk flow freely. In the past, I tried to use queue cards but always felt like they became a distraction and kept me from loosening-up on stage. By putting in the time and practicing I felt like I knew the slides without even looking at them, and this really helped increase my confidence.</p>
<h1 id="the-big-day">The big day</h1>
<p>On Saturday 8th June 2019, I presented my lightning talk to approximately 50 people at the Riddel Hall in Queens University, Belfast. Despite feeling super nervous, I was able to tackle my fears and deliver the talk. Everything went really smooth and the talk was well-received. Multiple other speakers and attendees congratulated me on my presentation and said they got something from the talk, which was incredible to hear.</p>
<p>For years, I’ve felt afraid to step out and face my fear of public speaking, but now, I feel like it’s only a matter of time until I take the next step to further push the boundaries. I’ll sign off with a quote that my wonderful girlfriend, Helen, reminded me of during my doubtful periods. Maybe it can help you, too.</p>
<p>“Never let the fear of striking out keep you from playing the game” - Babe Ruth</p>
<p><img src="../../../../images/2019-06-19-img2.jpg" alt="alt text" title="A photo of me feeling pretty relieved" /></p>
<p>-Andrew</p>After writing a post 6 months ago talking about how I wanted to blog once per week, I’ve realised that setting such a goal was ridiculous. Life gets busy and sometimes other priorities take precedent. Either way, I’m going to try and make a conscious effort to be more active in my writing here.Done is better than perfect2018-12-08T00:00:00+00:002018-12-08T00:00:00+00:00https://www.andrewhil.es/personal/2018/12/08/done-is-better-than-perfect<p>I’ve owned my domain, andrewhil.es, since 2013. It had always been my intention to create a website that would act as a personal journal where I could document my work, random thoughts and generally share articles on things I think others would benefit from knowing. However, shortly after I crafted my first incarnation of a holding page for the website, things didn’t progress much further.</p>
<h1 id="gathering-dust">Gathering dust</h1>
<p>Looking back, the last 5 years on this website haven’t been much more than a basic index.html file with nothing of great interest. I’d attempted various re-designs but never published anything as I could never quite get them over the line. Thankfully, my website gets around 100 hits a month so I don’t think anyone was overly disappointed when they stumbled upon my boring little corner of the web and found very little going on.</p>
<p>Each year I was renewing the hosting with good intentions but very little happened. I spend a large portion of my working day tweaking things in the browser and it’s a fundamental part of my occupation, to be able to craft fully-functional, production-ready websites. However, when it came to my own, it was just left, gathering dust. Perhaps a case of occupational hazard? Who knows.</p>
<h1 id="finding-new-motivation">Finding new motivation</h1>
<p>Back in October 2018 I FTP’d onto my web server for the first time in quite a while. I was looking for a few server configuration files when I came across all of the old content I’d created in previous years. Lots of different versions of holding pages, half-baked CSS files and images I’d snapped on my camera that never saw the light of day. In that moment, I quickly realised I’d forgotten about my online space for too long and it was time for a reboot.</p>
<p><img src="../../../../images/2018-12-08-img1.jpg" alt="alt text" title="Done is better than perfect poster image" /></p>
<p>In recent months, I’d listened to various podcasts where fellow web developers mentioned the curse of seeking perfection in their work and how it’s not always a positive. Make no mistake, we should always seek to deliver work of high quality, but perfection shouldn’t always be the level where we feel ready to show it off. In my mind, maybe that’s what has been holding me back from fixing this space and applying some effort to making it something worth caring for. I have a poster at home which lists lots of motivating quotes, one of which reads ‘Done is better than perfect’ - pretty fitting for this post.</p>
<p>With a shift in mindset, I realised that if I spent an hour here and there to try and spruce things up, eventually I’d get this site into a state where it’s fit enough for publishing to the web (not perfect, but a functioning work in progress).</p>
<p>Fast forward to today, Saturday 8th December 2018, and I’ve just finished giving this space a fresh lick of paint with some plain old CSS and it feels like the right time to deploy my changes. There’s no doubt, everything is still rough around the edges, but I know now that once I’ve hit the dreaded publish button and put it out for the world to see, that there’s an element of accountability placed upon me to maintain things and turn this site into something I’ll (hopefully) care about for many years to come.</p>
<h1 id="moving-forward">Moving forward</h1>
<p>Now that I’ve taken the first step and published something more than just a holding page, my aim is to continue enhancing this space. I’m still using a lot of older technologies under the hood, which brings an immediate impetus to remove them and modernize where I see fit. However, for now, I want to publish an article at least once a week. It’s ambitious, but without goals, I know I’ll struggle. So here we are. I hope you’ll come along for the ride.</p>
<p>-Andrew</p>I’ve owned my domain, andrewhil.es, since 2013. It had always been my intention to create a website that would act as a personal journal where I could document my work, random thoughts and generally share articles on things I think others would benefit from knowing. However, shortly after I crafted my first incarnation of a holding page for the website, things didn’t progress much further.