Analytics Snippet: Impersonating Scott Morrison and Bill Shorten

While the 2019 Australian Federal Election can be a very serious matter, the young Actuaries Data Analytics Group (yDAWG) decided to have a bit of fun. Using a Markov Chain generator and existing library of historical tweets, they explain how synthetic tweets can impersonate our political leaders.

For background on this data analytics adventure into the twittersphere, see this article where the yDAWG authors outline their investigation into what the 2019 candidates, parties and the public are saying about the election.

Let’s get started!

First, let’s load the packages. Note that markovify is the key package used in our exercise.

From their GitHub we can see that Markovify is a simple, extensible Markov chain generator. Right now, its primary use is for building Markov models of large corpora of text and generating random sentences from that. However, in theory, it could be used for other applications.

As a reminder, to install this package you just need to use the following command:

pip install markovify

In [51]:
import pandas as pd 
pd.set_option("display.max_colwidth", 200)
import warnings 
import markovify as mk
warnings.filterwarnings("ignore", category=DeprecationWarning)
 
Next, we read in the twitter data stored in a spreadsheet and convert it to a Pandas DataFrame.
In [52]:
sm = pd.read_excel(r'ScottMorrisonMP20190512-0710.xlsx')
bs = pd.read_excel('billshortenmp20190512-0716.xlsx')
 
There are about 3200 tweets for each candidate.
In [60]:
sm.shape, bs.shape
Out[60]:
((3242, 5), (3219, 5))
 

Let’s have a quick glance of the dataset.

In [56]:
sm.head()
Out[56]:
  text datetime username retweet_count
0 Our support for the first ever international netball hub highlights what can be done with our plan for a stronger economy and it recognises Melbourne’s reputation as a sporting city. 2019-05-11 06:50:44 ScottMorrisonMP 26
1 This new funding will see hundreds of change rooms upgraded across the country; secure the future of Australian netball; establish a permanent home & high performance centre for the @TheMatild… 2019-05-11 06:48:21 ScottMorrisonMP 29
2 Sport brings communities together. It also helps keep you active & healthy. Female athletes will be the main beneficiaries of our $70m commitment today a range of new sport initiatives, which … 2019-05-11 06:46:20 ScottMorrisonMP 42
3 My beautiful wife. Wherever we go, people want to chat to Jen, you can see why. https://t.co/Pits1kqRVC 2019-05-10 10:26:04 ScottMorrisonMP 149
4 We’re for jobs because jobs change lives. That’s why we’re committing to creating 1.25m more new jobs over the next 5 years.\n\n#BuildingOurEconomy https://t.co/tGYZfP9yGs 2019-05-09 21:37:13 ScottMorrisonMP 109
In [57]:
bs.head()
Out[57]:
  text datetime username retweet_count
0 If you love your ABC, if you want to reverse the Liberals’ cuts, if you want to protect it for the future, then I need your help. Vote Labor next Saturday to save the ABC. https://t.co/j9vzD0qzfu 2019-05-11 10:43:47 billshortenmp 654
1 Good doggos. https://t.co/dhF2z3B17w 2019-05-11 08:52:19 billshortenmp 201
2 LIVE from the Espy: Launching Labor’s National Cultural Policy – Creative Australia. https://t.co/IWCuAzsJWF 2019-05-11 05:13:42 billshortenmp 137
3 LIVE from Melbourne: Vote Labor to save the ABC https://t.co/DoGUhzowDl 2019-05-11 03:16:41 billshortenmp 253
4 LIVE from Melbourne: Vote Labor to save the ABC https://t.co/G7XzmXv0ZG 2019-05-11 02:32:10 billshortenmp 454
 

We will now add an extra column and concatenate the two dataframes

In [59]:
sm['candidate'] = 'Scott Morrison'
bs['candidate'] = 'Bill Shorten'
df = pd.concat([sm, bs])
df.head()
Out[59]:
  text datetime username retweet_count candidate
0 Our support for the first ever international netball hub highlights what can be done with our plan for a stronger economy and it recognises Melbourne’s reputation as a sporting city. 2019-05-11 06:50:44 ScottMorrisonMP 26 Scott Morrison
1 This new funding will see hundreds of change rooms upgraded across the country; secure the future of Australian netball; establish a permanent home & high performance centre for the @TheMatild… 2019-05-11 06:48:21 ScottMorrisonMP 29 Scott Morrison
2 Sport brings communities together. It also helps keep you active & healthy. Female athletes will be the main beneficiaries of our $70m commitment today a range of new sport initiatives, which … 2019-05-11 06:46:20 ScottMorrisonMP 42 Scott Morrison
3 My beautiful wife. Wherever we go, people want to chat to Jen, you can see why. https://t.co/Pits1kqRVC 2019-05-10 10:26:04 ScottMorrisonMP 149 Scott Morrison
4 We’re for jobs because jobs change lives. That’s why we’re committing to creating 1.25m more new jobs over the next 5 years.\n\n#BuildingOurEconomy https://t.co/tGYZfP9yGs 2019-05-09 21:37:13 ScottMorrisonMP 109 Scott Morrison
 

We confirmed that the number of tweets for each candidate remained as ~3200. We haven’t lost any data

In [61]:
df['candidate'].value_counts()
Out[61]:
Scott Morrison    3242
Bill Shorten      3219
Name: candidate, dtype: int64
 

Now, we define a function which does the following tasks:

  1. For a given candidate, read in all tweet texts, and store all texts into a single list
  2. Make a text model using the list
  3. Make 8 short tweets (which are of course fake)
In [62]:
def tweet(tweeter):
    doc = df[df['candidate']==tweeter].text.tolist()
    text_model = mk.Text(doc)
    print('\n', tweeter)
    for i in range(8):
        print(text_model.make_short_sentence(140))
In [35]:
gt;tweet('Scott Morrison')
tweet('Bill Shorten')
 
Scott Morrison
I will be keeping more of what you earn. That’s why today we’ve announced $78m to help those on lower incomes, on pensions…
These are the challenges being faced by the Archbishop http://t.co/6Wsj2SbwDt
Pensioners now better off under Coalition than they would not otherwise get #auspol
You can hear the full speech reported in Oz & will go up again next Friday
Let’s stick to my chat with Ray Hadley on @2GB873 this morning after 9am chatting to #RayHadley on @2GB873 #auspol https://t.co/jY4sVO7INw
Yesterday, Andrew Hastie & I have asked the team Wendy! https://t.co/dDK54Ezp7L
It was terrific to meet NZ Finance Minister Bill Morneau.
 
Bill Shorten
I’d rather give more tax to fund Malcolm Turnbull’s absurd definition of fairness: tax cuts – worth up to their extraordinary resilience.
You can now watch the launch of Labor’s vision for our netballers, this is never allowed to happen again. https://t.co/273NxItyY2
His broken promise will cost the average age of their standing care so much I made it an emoji.
Mr Turnbull says he supports these cuts. https://t.co/U9jUPDeGne It’s a simple vote in response. https://t.co/bDLSyJXGlO My message for #internationaldayofthegirl – we will eliminate trachoma from Australia by 2020.
Ravenclaw at a 70 year high – but he won’t try it again?
To find out how much they pay women compared to men.
The gender pay gap a big announcement on Medicare mean people are our boss and they did.
 

I will be keeping more of what you earn (by fake Scott Morrison)

The first fake tweet we made for Scott Morrison is really funny! I think it comes from this actual tweet:

We want you to keep more of what you earn (by real Scott Morrison)

Which is in row number 206 from our dataset.

In [65]:
sm['text'][206]
Out[65]:

Our Government has a strong record, clear beliefs and a comprehensive plan to keep our economy strong, keep Australians safe, and keep Australians together. We want you to keep more of what you earn. We don’t believe you should have to pull some people down to raise others up. https://t.co/2Y8Y7khCst’

 

Let’s try it again!

In [36]:
tweet('Scott Morrison')
tweet('Bill Shorten')
 
Scott Morrison
Meet Mick and Kris from @onthegolife and hear that UK-based Fintech companies are choosing Australia as was PM Wickremesinghe.
This has been awarded an OAM today for a stronger economy on the agenda at the Anzac Day Dawn Service http://t.co/80Fz1stF63
We will continue to keep more of what has been elected because the people’s trust in @liberalaus to manage… Transcript of media conference.
My focus is on the pension change and then cry crocodile tears when it comes to border protection. https://t.co/hKrbDe6bKi
The Turnbull Government has a problem with Nauruan population – not true
Bill Shorten should step up and gave her a hug.
Feds impose BasicsCard in SA to hear well-known Prince Harry fan Daphne Dunne has passed laws to close loopholes & ensure prof…
 
Bill Shorten
This election is a plan to do the same.
Labor will introduce a 15% GST. https://t.co/KmiyMiF9jm
Another discredited policy from a charity that feeds our vulnerable.
RT @RoveAndSam: Always a source of advice and inspiration. https://t.co/0NOT7YGDT7
He wants to end up in jail than to university.
And if you’re looking for a banking royal commission.
LIVE from Melbourne: Vote Labor next Saturday to save the ABC and keep it in public hands.
If your bank steals from you there’s no guarantee they’ll even get to do that with our Fair Go Action Pla…
 

Labor will introduce a 15% GST. https://t.co/KmiyMiF9jm‘ (by fake Bill Shorten)

If you click the link, you will see that the real Bill Shorten said

There is nothing fair about increasing the GST, @TurnbullMalcolm . @AustralianLabor stands against a 15% GST.

Another total twist of words! The scary thing is, how realistic the fake tweets look. For someone who doesn’t follow politics, the phrasing and content are pretty believable

Let’s now force the fake tweets to start from certain words.

Let’s start with the word Australia.

In [66]:
def subj_tweet(tweeter, subject):
    doc = df[df['candidate']==tweeter].text.tolist()
    text_model = mk.Text(doc) 
    print('\n', tweeter)
    for i in range(8):
        print(text_model.make_sentence_with_start(subject, strict=False))
In [40]:
subj_tweet('Scott Morrison', 'Australia')
subj_tweet('Bill Shorten', 'Australia')
 

Scott Morrison
Australia resettled almost 8000 refugee and humanitarian intake for 13/14 14/15 was from persecuted minorities, especially Christians.
Australia championed the cause of free and open heart will be a wake-up call for all directors, particularly those that are needed that is why the rot has set into Fed Labor – NSW Election 2015 – ABC News http://t.co/h6lFhW7UIc Australia pay their fair share of tax payers having to pay the…
Australia back on their border failures in immigration, and blown budget by $1.5 million over the past 21 days! Australia overachieving on our Government’s priority.
Australia for the latest NAB monthly survey. Australia has climbed back to top of carbon tax, then promises more subsidies – eventually.
Australia should not be included in the areas serviced by Mirabel have not been finally determined.

Bill Shorten

Australia doing it tough have benefited from the kids at risk of homelessness – women over 55.
Australia should know they are also with those here in WA and a relentless champion for his presence. https://t.co/dY2LKYXujP
Australia must do all we can achieve true equality for the launch of Soundtrack Australia, Labor’s policy to support my motion to never water down Australia’s gun laws.
Australia can be a bigot✅
Australia – those who risked and lost their lives in service of our whole nation | Bill Shorten https://t.co/IIfqmChGhA Australia lost 50,000 full-time jobs last year – we are bringing a new cardiac treatment theatre in Rockhampton, so locals get world class Paralympic athletes.
https://t.co/DYlBLv0T9a
Australia should be sacked or refused employment because of low wages growth. Australia where the gap wider.
 

Next, try Climate

In [42]:
subj_tweet('Scott Morrison', 'Climate')
subj_tweet('Bill Shorten', 'Climate')
 

Scott Morrison
Climate Solutions Fund to deliver more of what you have done for the economy to grow revenue.
Climate Solutions Fund to deliver a stronger position to fund the National Disability Insurance Scheme through a sea of blue in full song #upupcronulla http://t.co/UPXEdqOjeq
Climate Solutions Fund to deliver more of the future.
Climate Solutions Fund to deliver more jobs with details of policies one million more Aus…
Climate Solutions Fund to deliver a Budget surplus.
Climate Solutions Fund to deliver the inaugural @wesleymission Rev Dr Gordon Moyes Lecture @australian https://t.co/VbNGu7CgG8
Climate Solutions Fund to deliver a stronger economy is working – talking with them at a bit of re…
Climate Solutions Fund to deliver on Australia’s 2030 emissions reductions targets?

Bill Shorten
Climate March last night, we caught up in jail than to put One Nation to do that with our military personnel deployed in Kabul, Afghanistan.
Climate Change Action Plan will mean higher energy prices and pollution while scaring away investment and jobs than Turnbull’s $65 billion handout to the next election will be a YES voter for marriage equality today was vintage Swanny, a reminder that we intend to introduce a national scheme for safe, legal access to two years to improve, amend and pass these national security risks entry to Australia. https://t.co/SPgDFDaqdJ
Climate change, marriage equality, we need a Royal Commission.
Climate change on the banking sector https://t.co/MILz2qNki3
Climate change on the #MelbourneCup today
Climate Change Action Plan @Bowenchris …. https://t.co/ztwkyY7KUZ
Climate March last night, we caught up in gaol than university.
Climate March last night, we caught up in Question Time why his government should fund public schools, it’s that they’re choosing not to. https://t.co/SYy03FKWpJ

 

Budget

In [43]:
subj_tweet('Scott Morrison', 'Budget')
subj_tweet('Bill Shorten', 'Budget')
 

Scott Morrison
Budget surplus in 2021.
Budget makes the best country in the NT Matt Williams @TheNTNews https://t.co/9GkpMT46ax
Budget surplus in 2021.
Budget into the abuse of Australians https://t.co/B5CylSZ3iP
Budget surplus in 2019-20 & the Central Coast will be looking for work as part of the real progress but it is because it reminds me every single day transforming and transitioning our economy grew by 0.9% in Sept quarter – but we can’t take this growth for Australians https://t.co/fY5jxYxUNI
Budget today we would deliver jobs & increase building security in the attack.
Budget surplus in 2019-20 & the businesses that are used to write about how to make the case MORE https…
Budget focus on the Australian on Monday

Bill Shorten
Budget stronger in the view that when companies pay in @aflwomens https://t.co/y51oM0dcEp
Budget stronger in the @smh today. https://t.co/mJqh5bIifc
Budget Office costings, Labor’s plan has been constant: cuts to the independent Parliamentary Budget Office costings, Labor’s plan https://t.co/o2F5dzCc2X
Budget for big business and sells out workers.
Budget stronger in the Senate https://t.co/bm3k5lI6Vl
Budget Reply as Opposition Leader.
Budget stronger in the Liberal smear campaign against France.
Budget – with no justice.

 

‘Budget surplus in 2021’ got a couple of times. Scott Morrison seems really confident about this. OK, OK, I got it.

 

Taxes

In [44]:
subj_tweet('Scott Morrison', 'Taxes')
subj_tweet('Bill Shorten', 'Taxes')
 

Scott Morrison
Taxes as proposed by Labor – most significantly on our $3.5 Jobs for Families child care safety net
Taxes as proposed by Labor – most significantly on our borders secure and our commitment to competition law changes started today #auspol https://t.co/Y4QASi854k
Taxes as proposed by Labor on boats.
Taxes as proposed by Labor are standing in the world.
Taxes as proposed by Labor on what we’ve accomplished, become, still to do.
Taxes as proposed by Labor on what they pay in tax next year and she was just plain wrong to think others don’t Taxes as proposed by Labor – most significantly on our $3.5 Jobs for Families package – joined him today at #Brewarrina with the budget to balance #au…
Taxes as proposed by Labor – NSW Labor disease and to stay in work http://t.co/rkCqSmCkxT

Bill Shorten
None
None
None
None
None
None
None
None

 

Bill Shorten has nothing to comment on taxes…

 

I

In [45]:
subj_tweet('Scott Morrison', 'I')
subj_tweet('Bill Shorten', 'I')
 

Scott Morrison
I commend Presidents Trump & Xi for listening to farmers & seeing firsthand what the Sir John Monash Centre will be worth a read http://t.co/j94cbvPEzL
I delivered a tax cut come July. https://t.co/HNFaLdyiyh
I convened a meeting of the hard-working Australian businesses
I shared the story of Leslie “Bull” Allen, an ANZAC hero who is no solution | The Australian: http://t.co/pM8QF6NoL1
I thank the Council for their match ups today against Belarus for their service & extend our deepest condolences & to His Majesty Emperor Naruh…
I think taxes should be better off or unaffected by the Govt introduced measures to curb investor lending.
I helped her up and simplify trades.
I received during the #Wentworth campaign!

Bill Shorten
I laid out Labor’s positive vision… I seek to lead, ready to govern.
I walked past 17 year old gun @DylanAlcott taking an ice bath and, when I sat down with her extraordinary mind, her caring heart and her family every happiness in the aged pension in Australian history – it is a lack of diversity in the years to improve, amend and pass these laws as soon as possible.
I sat down with her extraordinary mind, her caring heart and mind to illuminating the lives it shaped.
I laid out Labor’s plan to cut all funding for 348,000 preschool kids.
I step into a hospital, I’m reminded of what Labor can be trusted with Medicare.
I still don’t like getting swooped by them.
I think it’s an extra $47 million in funding to help Australians prostate and metastatic cancers navigate their tough battle. https://t.co/Hofp64QUVq

I seek to lead, ready to govern. (by fake Bill Shorten)

Very poetic!

 

We

In [46]:
subj_tweet('Scott Morrison', 'We')
subj_tweet('Bill Shorten', 'We')
 

Scott Morrison
We say what happened was not a policy.
We stand with all New Zealanders as they do not need ‘gender whisperers’ in our lives and our economy. https://t.co/6cWBCMUGHy
We need a national energy guarantee will lower prices.
We began our response to the planned signing of a suit. https://t.co/hKKUstnArq
We just get on boats #ausvotes
We are turning around the new UK Chancellor Philip Hammond and Canadian Finance Minister @stevenljoyce.
We continue to build no naval vessels in Australia for the NDIS, which will include a new smart economy, so the benefits flow through our economy leads the EAO team for org Get Well Glenn Wheeler this morning at Australian Honey Products with @andrewnikolic https://t.co/31IHmfLAjo
We also talked trade & the businesses that employ 1/2 the Aus workforce – 6.5m workers just like the ABCC, that improve productivity. https://t.co/v5Ak0Ypm6T

Bill Shorten
We announced the detail on @billshortenmp’s plan to phase out single-use plastics can’t work without a job https://t.co/cLwlcFdTNR
We continue to push for an election can’t cover up for workers at Toll in Brissy about their pay, conditions + Labor’s plan to end the Medicare freeze within 50 days.
We honour his memory and offer our nation’s unfinished business.
We could have same-sex weddings this week if Mr Morrison voted against them.
We did this together. https://t.co/eUe6Tt7cJI
We champion the things to cut.
We don’t want our jobs and better wages, not bigger banks. https://t.co/7wdCsAuzAp
We announced that Labor’s cracking down on the marriage equality is to keep delivering for you and all our might #MelbourneCup

 

Great

In [50]:
subj_tweet('Scott Morrison', 'Great')
subj_tweet('Bill Shorten', 'Great')
 

Scott Morrison
Great fun at the Berwick Buddhist Temple, we paused to remember the victims of the sale of their proposals https://t.co/t4kgPzl9yC https://t.co/rEXjanJMdm
Great day on the front foot.
Great fun to have agreement on how to live well and was a fighter in campaigns and for causes.
Great questions on #Budget2016 https://t.co/WK4m3AKVVZ
Great group – even got a job in November alone.
Great fun to have you on a strong economic management, we can make my way there from the rebels!! Down 24-3 at half time and getting on with the MP for Gilmore @nyunggai https://t.co/X1xu4dML3c
Great if you could be created in 2017.
Great group – even got a plan to drive more and better together. https://t.co/GVNTq76714

Bill Shorten
Great story coming up tonight at 630 on SBS to begin talks on Treaty.
Great news: 90k Australians enrolled to vote. https://t.co/TWCBjI0MMW
Great game tonight – the fact that anyone survived those hellish conditions is a person in Australia – today Australia honours you.
Great piece on Mr Turnbull’s baseless scare campaign. https://t.co/jq6IYUJS2h
Great turnout at my 25th Town Hall https://t.co/1ihHVsbsi0
Great to be back at it, fighting for Medicare. https://t.co/uJ2873OQcu
Great news for local jobs!
Great piece on Mr Turnbull had plans to water down Labor’s laws against dodgy banking practices.

 

Conclusion

This exercise has proven to be a lot of fun. There are several things you can do to further improve the results:

  • remove special characters
  • use Spacy’s part of speech (POS) tagger to make better sounding sentences
  • get more data

 

To me, it’s very scary to see how easy it is to create a fake twitter bot. In the age of data analytics, all it needs is some skills to pull down a few thousand tweets, and spend half an hour coding time, thanks to the great Python open source community.

In fact fake tweets have already been used in the 2016 US election and had a great impact, according to ANU’s research. Something to think about next time you read stuff from the internet!

CPD: Actuaries Institute Members can claim two CPD points for every hour of reading articles on Actuaries Digital.

Comments

No comments.


Comment on the article (Be kind)

Your comment will be revised by the site if needed.