Analytics Snippet: Impersonating Scott Morrison and Bill Shorten
While the 2019 Australian Federal Election can be a very serious matter, the young Actuaries Data Analytics Group (yDAWG) decided to have a bit of fun. Using a Markov Chain generator and existing library of historical tweets, they explain how synthetic tweets can impersonate our political leaders.
For background on this data analytics adventure into the twittersphere, see this article where the yDAWG authors outline their investigation into what the 2019 candidates, parties and the public are saying about the election.
Let’s get started!
First, let’s load the packages. Note that markovify is the key package used in our exercise.
From their GitHub we can see that Markovify is a simple, extensible Markov chain generator. Right now, its primary use is for building Markov models of large corpora of text and generating random sentences from that. However, in theory, it could be used for other applications.
As a reminder, to install this package you just need to use the following command:
pip install markovify
import pandas as pd pd.set_option("display.max_colwidth", 200) import warnings import markovify as mk warnings.filterwarnings("ignore", category=DeprecationWarning)
sm = pd.read_excel(r'ScottMorrisonMP20190512-0710.xlsx') bs = pd.read_excel('billshortenmp20190512-0716.xlsx')
((3242, 5), (3219, 5))
Let’s have a quick glance of the dataset.
|0||Our support for the first ever international netball hub highlights what can be done with our plan for a stronger economy and it recognises Melbourne’s reputation as a sporting city.||2019-05-11 06:50:44||ScottMorrisonMP||26|
|1||This new funding will see hundreds of change rooms upgraded across the country; secure the future of Australian netball; establish a permanent home & high performance centre for the @TheMatild…||2019-05-11 06:48:21||ScottMorrisonMP||29|
|2||Sport brings communities together. It also helps keep you active & healthy. Female athletes will be the main beneficiaries of our $70m commitment today a range of new sport initiatives, which …||2019-05-11 06:46:20||ScottMorrisonMP||42|
|3||My beautiful wife. Wherever we go, people want to chat to Jen, you can see why. https://t.co/Pits1kqRVC||2019-05-10 10:26:04||ScottMorrisonMP||149|
|4||We’re for jobs because jobs change lives. That’s why we’re committing to creating 1.25m more new jobs over the next 5 years.\n\n#BuildingOurEconomy https://t.co/tGYZfP9yGs||2019-05-09 21:37:13||ScottMorrisonMP||109|
|0||If you love your ABC, if you want to reverse the Liberals’ cuts, if you want to protect it for the future, then I need your help. Vote Labor next Saturday to save the ABC. https://t.co/j9vzD0qzfu||2019-05-11 10:43:47||billshortenmp||654|
|1||Good doggos. https://t.co/dhF2z3B17w||2019-05-11 08:52:19||billshortenmp||201|
|2||LIVE from the Espy: Launching Labor’s National Cultural Policy – Creative Australia. https://t.co/IWCuAzsJWF||2019-05-11 05:13:42||billshortenmp||137|
|3||LIVE from Melbourne: Vote Labor to save the ABC https://t.co/DoGUhzowDl||2019-05-11 03:16:41||billshortenmp||253|
|4||LIVE from Melbourne: Vote Labor to save the ABC https://t.co/G7XzmXv0ZG||2019-05-11 02:32:10||billshortenmp||454|
We will now add an extra column and concatenate the two dataframes
sm['candidate'] = 'Scott Morrison' bs['candidate'] = 'Bill Shorten' df = pd.concat([sm, bs]) df.head()
|0||Our support for the first ever international netball hub highlights what can be done with our plan for a stronger economy and it recognises Melbourne’s reputation as a sporting city.||2019-05-11 06:50:44||ScottMorrisonMP||26||Scott Morrison|
|1||This new funding will see hundreds of change rooms upgraded across the country; secure the future of Australian netball; establish a permanent home & high performance centre for the @TheMatild…||2019-05-11 06:48:21||ScottMorrisonMP||29||Scott Morrison|
|2||Sport brings communities together. It also helps keep you active & healthy. Female athletes will be the main beneficiaries of our $70m commitment today a range of new sport initiatives, which …||2019-05-11 06:46:20||ScottMorrisonMP||42||Scott Morrison|
|3||My beautiful wife. Wherever we go, people want to chat to Jen, you can see why. https://t.co/Pits1kqRVC||2019-05-10 10:26:04||ScottMorrisonMP||149||Scott Morrison|
|4||We’re for jobs because jobs change lives. That’s why we’re committing to creating 1.25m more new jobs over the next 5 years.\n\n#BuildingOurEconomy https://t.co/tGYZfP9yGs||2019-05-09 21:37:13||ScottMorrisonMP||109||Scott Morrison|
We confirmed that the number of tweets for each candidate remained as ~3200. We haven’t lost any data
Scott Morrison 3242 Bill Shorten 3219 Name: candidate, dtype: int64
Now, we define a function which does the following tasks:
- For a given candidate, read in all tweet texts, and store all texts into a single list
- Make a text model using the list
- Make 8 short tweets (which are of course fake)
def tweet(tweeter): doc = df[df['candidate']==tweeter].text.tolist() text_model = mk.Text(doc) print('\n', tweeter) for i in range(8): print(text_model.make_short_sentence(140))
gt;tweet('Scott Morrison') tweet('Bill Shorten')
I will be keeping more of what you earn (by fake Scott Morrison)
The first fake tweet we made for Scott Morrison is really funny! I think it comes from this actual tweet:
We want you to keep more of what you earn (by real Scott Morrison)
Which is in row number 206 from our dataset.
Our Government has a strong record, clear beliefs and a comprehensive plan to keep our economy strong, keep Australians safe, and keep Australians together. We want you to keep more of what you earn. We don’t believe you should have to pull some people down to raise others up. https://t.co/2Y8Y7khCst’
Let’s try it again!
tweet('Scott Morrison') tweet('Bill Shorten')
Labor will introduce a 15% GST. https://t.co/KmiyMiF9jm‘ (by fake Bill Shorten)
If you click the link, you will see that the real Bill Shorten said
There is nothing fair about increasing the GST, @TurnbullMalcolm . @AustralianLabor stands against a 15% GST.
Another total twist of words! The scary thing is, how realistic the fake tweets look. For someone who doesn’t follow politics, the phrasing and content are pretty believable
Let’s now force the fake tweets to start from certain words.
Let’s start with the word Australia.
def subj_tweet(tweeter, subject): doc = df[df['candidate']==tweeter].text.tolist() text_model = mk.Text(doc) print('\n', tweeter) for i in range(8): print(text_model.make_sentence_with_start(subject, strict=False))
subj_tweet('Scott Morrison', 'Australia') subj_tweet('Bill Shorten', 'Australia')
Australia resettled almost 8000 refugee and humanitarian intake for 13/14 14/15 was from persecuted minorities, especially Christians.
Australia championed the cause of free and open heart will be a wake-up call for all directors, particularly those that are needed that is why the rot has set into Fed Labor – NSW Election 2015 – ABC News http://t.co/h6lFhW7UIc Australia pay their fair share of tax payers having to pay the…
Australia back on their border failures in immigration, and blown budget by $1.5 million over the past 21 days! Australia overachieving on our Government’s priority.
Australia for the latest NAB monthly survey. Australia has climbed back to top of carbon tax, then promises more subsidies – eventually.
Australia should not be included in the areas serviced by Mirabel have not been finally determined.
Next, try Climate
subj_tweet('Scott Morrison', 'Climate') subj_tweet('Bill Shorten', 'Climate')
Climate Solutions Fund to deliver more of what you have done for the economy to grow revenue.
Climate Solutions Fund to deliver a stronger position to fund the National Disability Insurance Scheme through a sea of blue in full song #upupcronulla http://t.co/UPXEdqOjeq
Climate Solutions Fund to deliver more of the future.
Climate Solutions Fund to deliver more jobs with details of policies one million more Aus…
Climate Solutions Fund to deliver a Budget surplus.
Climate Solutions Fund to deliver the inaugural @wesleymission Rev Dr Gordon Moyes Lecture @australian https://t.co/VbNGu7CgG8
Climate Solutions Fund to deliver a stronger economy is working – talking with them at a bit of re…
Climate Solutions Fund to deliver on Australia’s 2030 emissions reductions targets?
Climate March last night, we caught up in jail than to put One Nation to do that with our military personnel deployed in Kabul, Afghanistan.
Climate Change Action Plan will mean higher energy prices and pollution while scaring away investment and jobs than Turnbull’s $65 billion handout to the next election will be a YES voter for marriage equality today was vintage Swanny, a reminder that we intend to introduce a national scheme for safe, legal access to two years to improve, amend and pass these national security risks entry to Australia. https://t.co/SPgDFDaqdJ
Climate change, marriage equality, we need a Royal Commission.
Climate change on the banking sector https://t.co/MILz2qNki3
Climate change on the #MelbourneCup today
Climate Change Action Plan @Bowenchris …. https://t.co/ztwkyY7KUZ
Climate March last night, we caught up in gaol than university.
Climate March last night, we caught up in Question Time why his government should fund public schools, it’s that they’re choosing not to. https://t.co/SYy03FKWpJ
subj_tweet('Scott Morrison', 'Budget') subj_tweet('Bill Shorten', 'Budget')
Budget surplus in 2021.
Budget makes the best country in the NT Matt Williams @TheNTNews https://t.co/9GkpMT46ax
Budget surplus in 2021.
Budget into the abuse of Australians https://t.co/B5CylSZ3iP
Budget surplus in 2019-20 & the Central Coast will be looking for work as part of the real progress but it is because it reminds me every single day transforming and transitioning our economy grew by 0.9% in Sept quarter – but we can’t take this growth for Australians https://t.co/fY5jxYxUNI
Budget today we would deliver jobs & increase building security in the attack.
Budget surplus in 2019-20 & the businesses that are used to write about how to make the case MORE https…
Budget focus on the Australian on Monday
Budget stronger in the view that when companies pay in @aflwomens https://t.co/y51oM0dcEp
Budget stronger in the @smh today. https://t.co/mJqh5bIifc
Budget Office costings, Labor’s plan has been constant: cuts to the independent Parliamentary Budget Office costings, Labor’s plan https://t.co/o2F5dzCc2X
Budget for big business and sells out workers.
Budget stronger in the Senate https://t.co/bm3k5lI6Vl
Budget Reply as Opposition Leader.
Budget stronger in the Liberal smear campaign against France.
Budget – with no justice.
‘Budget surplus in 2021’ got a couple of times. Scott Morrison seems really confident about this. OK, OK, I got it.
subj_tweet('Scott Morrison', 'Taxes') subj_tweet('Bill Shorten', 'Taxes')
Taxes as proposed by Labor – most significantly on our $3.5 Jobs for Families child care safety net
Taxes as proposed by Labor – most significantly on our borders secure and our commitment to competition law changes started today #auspol https://t.co/Y4QASi854k
Taxes as proposed by Labor on boats.
Taxes as proposed by Labor are standing in the world.
Taxes as proposed by Labor on what we’ve accomplished, become, still to do.
Taxes as proposed by Labor on what they pay in tax next year and she was just plain wrong to think others don’t Taxes as proposed by Labor – most significantly on our $3.5 Jobs for Families package – joined him today at #Brewarrina with the budget to balance #au…
Taxes as proposed by Labor – NSW Labor disease and to stay in work http://t.co/rkCqSmCkxT
Bill Shorten has nothing to comment on taxes…
subj_tweet('Scott Morrison', 'I') subj_tweet('Bill Shorten', 'I')
I commend Presidents Trump & Xi for listening to farmers & seeing firsthand what the Sir John Monash Centre will be worth a read http://t.co/j94cbvPEzL
I delivered a tax cut come July. https://t.co/HNFaLdyiyh
I convened a meeting of the hard-working Australian businesses
I shared the story of Leslie “Bull” Allen, an ANZAC hero who is no solution | The Australian: http://t.co/pM8QF6NoL1
I thank the Council for their match ups today against Belarus for their service & extend our deepest condolences & to His Majesty Emperor Naruh…
I think taxes should be better off or unaffected by the Govt introduced measures to curb investor lending.
I helped her up and simplify trades.
I received during the #Wentworth campaign!
I laid out Labor’s positive vision… I seek to lead, ready to govern.
I walked past 17 year old gun @DylanAlcott taking an ice bath and, when I sat down with her extraordinary mind, her caring heart and her family every happiness in the aged pension in Australian history – it is a lack of diversity in the years to improve, amend and pass these laws as soon as possible.
I sat down with her extraordinary mind, her caring heart and mind to illuminating the lives it shaped.
I laid out Labor’s plan to cut all funding for 348,000 preschool kids.
I step into a hospital, I’m reminded of what Labor can be trusted with Medicare.
I still don’t like getting swooped by them.
I think it’s an extra $47 million in funding to help Australians prostate and metastatic cancers navigate their tough battle. https://t.co/Hofp64QUVq
I seek to lead, ready to govern. (by fake Bill Shorten)
subj_tweet('Scott Morrison', 'We') subj_tweet('Bill Shorten', 'We')
We say what happened was not a policy.
We stand with all New Zealanders as they do not need ‘gender whisperers’ in our lives and our economy. https://t.co/6cWBCMUGHy
We need a national energy guarantee will lower prices.
We began our response to the planned signing of a suit. https://t.co/hKKUstnArq
We just get on boats #ausvotes
We are turning around the new UK Chancellor Philip Hammond and Canadian Finance Minister @stevenljoyce.
We continue to build no naval vessels in Australia for the NDIS, which will include a new smart economy, so the benefits flow through our economy leads the EAO team for org Get Well Glenn Wheeler this morning at Australian Honey Products with @andrewnikolic https://t.co/31IHmfLAjo
We also talked trade & the businesses that employ 1/2 the Aus workforce – 6.5m workers just like the ABCC, that improve productivity. https://t.co/v5Ak0Ypm6T
We announced the detail on @billshortenmp’s plan to phase out single-use plastics can’t work without a job https://t.co/cLwlcFdTNR
We continue to push for an election can’t cover up for workers at Toll in Brissy about their pay, conditions + Labor’s plan to end the Medicare freeze within 50 days.
We honour his memory and offer our nation’s unfinished business.
We could have same-sex weddings this week if Mr Morrison voted against them.
We did this together. https://t.co/eUe6Tt7cJI
We champion the things to cut.
We don’t want our jobs and better wages, not bigger banks. https://t.co/7wdCsAuzAp
We announced that Labor’s cracking down on the marriage equality is to keep delivering for you and all our might #MelbourneCup
subj_tweet('Scott Morrison', 'Great') subj_tweet('Bill Shorten', 'Great')
Great fun at the Berwick Buddhist Temple, we paused to remember the victims of the sale of their proposals https://t.co/t4kgPzl9yC https://t.co/rEXjanJMdm
Great day on the front foot.
Great fun to have agreement on how to live well and was a fighter in campaigns and for causes.
Great questions on #Budget2016 https://t.co/WK4m3AKVVZ
Great group – even got a job in November alone.
Great fun to have you on a strong economic management, we can make my way there from the rebels!! Down 24-3 at half time and getting on with the MP for Gilmore @nyunggai https://t.co/X1xu4dML3c
Great if you could be created in 2017.
Great group – even got a plan to drive more and better together. https://t.co/GVNTq76714
Great story coming up tonight at 630 on SBS to begin talks on Treaty.
Great news: 90k Australians enrolled to vote. https://t.co/TWCBjI0MMW
Great game tonight – the fact that anyone survived those hellish conditions is a person in Australia – today Australia honours you.
Great piece on Mr Turnbull’s baseless scare campaign. https://t.co/jq6IYUJS2h
Great turnout at my 25th Town Hall https://t.co/1ihHVsbsi0
Great to be back at it, fighting for Medicare. https://t.co/uJ2873OQcu
Great news for local jobs!
Great piece on Mr Turnbull had plans to water down Labor’s laws against dodgy banking practices.
This exercise has proven to be a lot of fun. There are several things you can do to further improve the results:
- remove special characters
- use Spacy’s part of speech (POS) tagger to make better sounding sentences
- get more data
To me, it’s very scary to see how easy it is to create a fake twitter bot. In the age of data analytics, all it needs is some skills to pull down a few thousand tweets, and spend half an hour coding time, thanks to the great Python open source community.
In fact fake tweets have already been used in the 2016 US election and had a great impact, according to ANU’s research. Something to think about next time you read stuff from the internet!
CPD: Actuaries Institute Members can claim two CPD points for every hour of reading articles on Actuaries Digital.