Background
AU tech jobs is a job aggregator and application tracking system for tech jobs in AU mainly focused on companies that provide work visa sponsorship. It is a job aggregator website/app that lists jobs from various sources of any careers' page from this list. Rather than going to 50+ websites to find a job that matches your skills you can look it up at a single place. --- Usecases Rather than visiting 45-50 career pages looking for jobs the customer can get all the jobs from most AU companies that provide work visa sponsorship. The two main users are as follows: Students in AU in their final semester Finding a full time tech job in AU is not an easy task. Even after studying 2 years masters in IT it is still difficult. On top of it many people don't know that there are companies that sponsor work visa (visa 482) that can lead to a Permanent Residency (PR) if you stay with the same employer on a similar position for 3 years. This website will list all potential employers that can sponsor work visa. Skilled people outside Australia Current trend for PR in AU for IT is minimum at 95 points which requires a lot of work, at end of 2018 it was at 70-75 points. This website can act like a place to get all the tech jobs in one place, follow them, apply for them, and track applications for individual job seekers that are outside of AU right now. Technical Decisions This should be a 3 part application, with following parts: Scraper It can be written in Python Scrapy, there will be a distinct scraper for each website we scrape. All data will be of same format from each site we scrape. The scraper part will write the data to a database (option of nosql db is also open). This part will entail having a great db design too. New jobs will be added every day or every 12 hours by the scarper and it will hide expired jobs on the scape run. Scraper can in node too. We might need 20+ different scarper going into 50 odd sites, we will need to fetch new companies from Stackoverflow too. Handling duplicates Duplicate jobs should not be inserted in the database. Like same job listed in 2 recruitment company websites. Updated jobs might be updated in the database. Idempotent The scraper needs to be idempotent meaning running the scraper once, twice or 100 times should not mess up the data. It can also have a date parameter of from and to. Like if the scraper ran successfully on 10-Jan-2020 at 1 AM then we should be able to ask the scraper to get data only after the last successful run of 10-Jan-2020 1:00 AM. Backend It can be written in NodeJs where it will query the data from the datasource a relatiional db and spit out JSON. Frontend It can be a light ExpressJs (NodeJs) app itself just to save yourself from all the prerender headache and SEO tweaks. We can have a look at a react frontend option too. Other frontends can be a mobile app with ReactNative.