Machine Reading IMF Data: Data Retrieval with Python 

Updated November 16, 2016

Introduction

The International Monetary Fund (IMF) Statistics Department (STA) allows API access to their economic time series. Well-known datasets such as International Financial Statistics (IFS) can be machine read through the API. This example will use Python to retrieve Direction of Trade Statistics (DOTS) data from STA’s JSON RESTful Web Service so that we can determine the United States’ share of world exports over the past 50 years.

The IMF knowledge base provides more information on the three avaiable API formats and IMF data services. For more information on the work of STA, see their PDF annual report (PDF), STA at a glance 2015.

Gathering series and dimension information

First, we will need to import the requests and pandas libraries. These will allow us to read json data, open urls, and request information from the web.

Libraries

In [1]:
# Import libraries
import requests
import pandas as pd

Since we are using the JSON RESTful API, we start by using the ‘Dataflow’ endpoint URL to look at what series are available and find the series id of interest. The full output is long, so I’ve removed the data unrelated to this example. The IMF has many more series than what is shown below.

Find Series Name

In [2]:
# Find the series id and text name.
url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/Dataflow/'
seriesids = requests.get(url).json()
df = pd.DataFrame(seriesids['Structure']['Dataflows']['Dataflow'])
for x in range(6, 13):
     items = (str(df['@id'][x]), str(df['Name'][x]['#text']))
     print ': '.join(items)
DOT: Direction of Trade Statistics (DOTS)
FSIREM: Financial Soundness Indicators (FSI), Reporting Entities - Multidimensional
CDIS: Coordinated Direct Investment Survey (CDIS)
GFS01M: Government Finance Statistics (GFS 2001) - Multidimensional
GFS01: Government Finance Statistics (GFS 2001)
BOP: Balance of Payments (BOP)
BOPAGG: Balance of Payments (BOP), World and Regional Aggregates

We found above that the id for Direction of Trade Statistics is DOT. We can use this id to read notes about the series. We will next need to identify the dimensions of the data. For example, direction of trade data is based on a home country a flow and measure and a counterpart country. The data also has multiple frequencies and units of measurement. All of this information will be needed to later make our data request.

Find Series Details and Description

In [3]:
# Annotations for the series
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/DataStructure/DOT"
dotstruct = requests.get(url).json()
df = pd.DataFrame(dotstruct['Structure']['KeyFamilies']\
     ['KeyFamily']['Annotations'])
for x in range(0, 7): 
     items = (str(df['Annotation'][x]['AnnotationTitle']), \
     str(df['Annotation'][x]['AnnotationText']['#text']))
     print ': '.join(items)
Latest Update Date: 04/26/2016
Name: Direction of Trade Statistics (DOTS)
Temporal Coverage: Monthly and quarterly data are available starting 1960. Annual data are available starting 1947.
Geographic Coverage: DOTS covers 184 countries, the world, and major areas.
Methodology: Guide to Direction of Trade Statistics, 1993. See Documents tab.
Definition: The <B>Direction of Trade Statistics (DOTS)</B> presents current figures on the value of merchandise exports and imports disaggregated according to a country's primary trading partners. Area and world aggregates are included in the display of trade flows between major areas of the world. Reported data is supplemented by estimates whenever such data is not available or current. Imports are reported on a cost, insurance and freight (CIF) basis and exports are reported on a free on board (FOB) basis, with the exception of a few countries for which imports are also available FOB. Time series data includes estimates derived from reports of partner countries for non-reporting and slow-reporting countries.
Code: DOT

Find Series Dimensions

In [4]:
# Look at structure of DOTS data to find the dimensions for our data request
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/DataStructure/DOT"
dotstruct = requests.get(url).json()
df = pd.DataFrame(dotstruct['Structure']['KeyFamilies']['KeyFamily']\
          ['Components']['Dimension'])
for x in range(0, 4): 
     items = ("Dimension", str(x+1), str(df['@codelist'][x]))
     print ': '.join(items)
Dimension: 1: CL_FREQ
Dimension: 2: CL_AREA_DOT
Dimension: 3: CL_INDICATOR_DOT
Dimension: 4: CL_COUNTERPART_AREA_DOT

We can now copy the code for each dimension into the CodeList Method to get the list of possible values. For example, we will need to identify the value of the second dimension, CL_AREA_DOT, for the United States. Below, we show that the code is US. I’ve manually placed the index number for the U.S. and World codes (again to save space), however, you can replace [200, 248] with [0, 248] to get the full list of country/area codes.

Find Country Codes

In [5]:
# Obtain country codes
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/CodeList/CL_AREA_DOT"
country = requests.get(url).json()
df = pd.DataFrame(country['Structure']['CodeLists']['CodeList']['Code'])
for x in [200, 248]: 
     items = (str(df['@value'][x]), str(df['Description'][x]['#text']))
     print ': '.join(items)
US: United States
W00: All Countries, excluding the IO

The series ID is DOT and the country codes (we will use this with the exporting country, CL_AREA_DOT, and the counterpart, CL_COUNTERPART_AREA_DOT) of interest are W00 for world and US for the US. We see below that the indicator of interest is TXG_FOB_USD, Goods, Value of Exports, Free on board (FOB), US Dollars.

Find Column IDs

In [6]:
# Obtain series info and ids
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/CodeList/CL_INDICATOR_DOT"
series = requests.get(url).json()
df = pd.DataFrame(series['Structure']['CodeLists']['CodeList']['Code'])
for x in range(0, 4): 
     items = (str(df['@value'][x]), str(df['Description'][x]['#text']))
     print ': '.join(items)
TXG_FOB_USD: Goods, Value of Exports, Free on board (FOB), US Dollars
TMG_CIF_USD: Goods, Value of Imports, Cost, Insurance, Freight (CIF), US Dollars
TMG_FOB_USD: Goods, Value of Imports, Free on board (FOB), US Dollars
All Indicators: All Indicators

We repeat the above steps for each dimension and record which series values are of interest to us.

Retrieving Data

The guide to STA’s API shows how we can combine information from the previous steps to call and retrieve data. For direction of trade statistics, we see that the dimensions are as follows:

  • Dimension 1: CL_FREQ (the frequency of the data–we want to use monthly data) – M
  • Dimension 2: CL_AREA_DOT (the primary country) – US
  • Dimension 3: CL_INDICATOR_DOT (the measure–we want to look at exports free of board) – TXG_FOB_USD
  • Dimension 4: CL_COUNTERPART_AREA_DOT (the counterpart country) – W00

The JSON RESTful API method for requesting the data is the CompactData Method. The format for putting together dimension and time period information is shown on the Web Service knowledge base as:

http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/{database ID}/ {item1 from dimension1}+{item2 from dimension1}{item N from dimension1}.{item1 from dimension2} +{item2 from dimension2}+{item M from dimension2}? startPeriod={start date}&endPeriod={end date}

Putting all of this information together, the URL to retrieve a JSON dictionary for 1966-2016 US exports to the world data is:

http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US.TXG_FOB_USD.W00.?startPeriod=1981&endPeriod=2016

The python code which gets the data and saves it as a dictionary is as follows:

Request data from IMF API

In [7]:
url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US.TXG_FOB_USD.W00.?startPeriod=1981&endPeriod=2016'
data = requests.get(url).json()
usexp = pd.DataFrame(data['CompactData']['DataSet']['Series']['Obs'])
usexp.columns = ['status', 'usexports','date'];
usexp.tail()
Out[7]:
date usexports
415 2015-08 123065777734
416 2015-09 125394024247
417 2015-10 130599515853
418 2015-11 120731632371
419 2015-12 119907169367

We can repeat the above code with a different URL to obtain data on total world exports and the exports of other countries which we may want to compare with the United States. We combine the request for several series into one URL, by adding ‘+code2+code3’. For example, ‘US+JP+CN.TXG..’

Example of request with multiple columns

In [8]:
ourl = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US+CN+JP+W00.TXG_FOB_USD.W00.?startPeriod=1972&endPeriod=2016'
odata = requests.get(ourl).json();

Cleaning the dataframe and naming rows

In [9]:
wexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][0]['Obs'])
wexp.columns = ['status','wexports','date']
del wexp['date']
del wexp['status']
chexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][1]['Obs'])
chexp.columns = ['status', 'chexports','date']
del chexp['date']
del chexp['status']
jpexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][2]['Obs'])
jpexp.columns = ['jpexports','date']
del jpexp['date']
usexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][3]['Obs'])
usexp.columns = ['status', 'usexports','date']
del usexp['status'];

Now we combine the two series into one dataframe and tell our script to read the export value columns as numbers.

Read as numeric

In [10]:
combined = pd.concat([usexp, wexp, chexp, jpexp], axis=1)
pd.to_datetime(combined.date)
combined = combined.set_index(pd.DatetimeIndex(combined['date']))
usexports = pd.to_numeric(combined.usexports)
wexports = pd.to_numeric(combined.wexports)
cexports = pd.to_numeric(combined.chexports)
jexports = pd.to_numeric(combined.jpexports)

Finally, we can calculate the U.S. percentage share of world exports. We simply divide the us exports by the world exports and multiply by 100. If using the data for economic research, we would likely take the log forms and apply some filters.

Capture

Calculate share of world exports for each country

In [11]:
combined['usshare'] = usexports / wexports * 100
combined['chinashare'] = cexports / wexports * 100
combined['japanshare'] = jexports / wexports * 100
combined.tail()

Out[11]:

date usshare chinashare japanshare
2015-08 9.460 15.121 3.668
2015-09 8.830 14.455 3.754
2015-10 9.330 13.737 3.892
2015-11 9.018 14.663 3.645
2015-12 8.776 16.362 3.804

Graphing the data

Let’s use matplotlib to view the result of our work.

Graph of US share of world exports

In [12]:
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
txt = '''Source: International Monetary Fund.'''

# Plot US share of world exports
combined.usshare.plot(grid=True, figsize=(9, 5), color="blue", linewidth=2,)
plt.ylabel('percentage of world exports')
plt.xlabel('Year')
plt.text(20,4.5,txt)
plt.title('U.S. Share of World Exports');
download

The graph shows a decrease in the U.S. share of exports from nearly 20 percent in 1966 to roughly 9 percent in 2015. We can also easily examine how changes in the U.S. share of exports compare with changes in the share of Japan and China.

Graph of moving average of US, China, and Japan shares of world exports

In [13]:
# Calculate moving average for each share, to reduce noise
combined['ussharema'] = combined['usshare'].rolling(12,12).mean()
combined['chsharema'] = combined['chinashare'].rolling(12,12).mean()
combined['jpsharema'] = combined['japanshare'].rolling(12,12).mean()

combshares = combined[['ussharema', 'chsharema', 'jpsharema']]
shares = list(combshares);
# Plot various shares of world exports
combined[shares][120:].plot(grid=True, figsize=(9, 5), linewidth=2)
plt.ylabel('percentage of world exports')
plt.xlabel('Year')
plt.text(150,-2,txt)
plt.title('Share of World Exports', );
download (1)

Export dataset to .csv

Let’s save the dataset in a portable format that can be read by any statistical software. My preference is to create a .csv file, which I will use for my U.S. Macroeconomic and Markets Dashboard.

Create csv file

In [14]:
combined.to_csv('us_share_exports.csv')

 

Advertisements

Dashboard update: Inaction has reactions

Dashboard PDF file:

Macro and Markets Dashboard: United States (April 30, 2016 — PDF)

Dashboard update summary:

Markets closed down slightly on the week following inaction from both the Fed and Bank of Japan. Oil and Yen both still became five percent more expensive in dollar terms. Advance estimate 2016 Q1 real GDP growth was weak at 0.6 percent, but in line with expectations. Labor markets continue to be a bright spot, with eyes on next Friday’s April jobs report.

rgdpgrowth_apr302016

Real GDP ticked up 0.6 percent in log terms in the first quarter of 2016. Personal consumption expenditures and residential fixed investment helped to keep GDP growth positive despite a decrease in nonresidential fixed and inventory investment, and growth in the trade gap. The personal savings rate also increased slightly, in quarterly terms, to 5.2 percent in Q1 from 5.0 percent the previous quarter.

FOMC meeting statement changes are kindly highlighted by the Wall Street Journal and include removal of the language about global risks and, some suggestion in my view, based on labor market growth, household income, and consumer sentiment, of a June rate hike. This would depend on continued labor market strength and price pressure plus an upward revision of Q1 GDP.

pcecomp_apr302016

Personal consumption expenditures (PCE) continued to increase in Q1, led by higher spending on services. However, monthly data on PCE as a share of GDP decreased slightly in March over its February level. PCE on durable goods as a percentage of GDP was also down slightly in March, to 7.3 percent from 7.4 percent in February.

Labor market data continues to be spotless. The weekly 257,000 new jobless claims is still near the multi-decade low. Next Friday is jobs day. Both the labor force participation rate and wages should continue to improve.

nasdaq_apr302016

Disappointing earnings data from Apple includes the first decline in iPhone sales nearly since its introduction and a slowdown in sales in China. This hit the Nasdaq particularly hard, as the composite index was down 2.7 percent while the S&P 500 and Dow Jones industrial average fell 1.3 percent each during the week.

oil_apr302016

Oil prices still managed to climb five percent over the past five days. West Texas Intermediate (WTI) crude oil was trading above $46 a barrel at several points during the week. Two recent stories reminded me how oil price fluctuations cause enormous transfers of wealth between countries. Jamaica was praised in this week’s Alphachat series on sovereign debt, while noting that their recent fiscal fortune is aligned with lower prices on their fuel imports. Likewise, a recent IMF publication noted that oil exporting economies in the middle east and central Asia have enacted powerful fiscal stimulus measures to keep their economies moving while they suffer the oil revenue slowdown. Those who believe in the resource curse might note that government measures to shift the economy away from oil are both important and difficult to achieve.

Lastly, the Yen had a volatile week, closing with a five percent appreciation against the dollar. The Brazilian real appreciated nearly four percent during the week, the Swiss franc and Turkish lira appreciated nearly two percent each, the pound sterling nearly one and a half percent and the Canadian dollar nearly one percent.

The full dashboard is here: Macro and Markets Dashboard: United States (April 30, 2016 — PDF)

Dashboard update: oil rally

Dashboard PDF file:

Macro and Markets Dashboard: United States (April 23, 2016 — PDF)

Dashboard update summary:

Oil prices continue to play a dominant role in other markets’ movements. As mentioned in previous updates, corporate earnings expectations are depressed, so even dismal performance (GS Q1 earnings fell 60% for example) can lead individual stocks higher. However, recovery in energy and commodity markets pushes investors from uncertainty-safe investments into riskier pools. This past week was relatively quiet on the macroeconomic data side, and market volatility was low, however, next week offers both Q1 GDP first estimates and an FOMC meeting, and volatility may tick back up.

While Doha talks set expectations for a fall in oil prices, markets actually pushed prices up by 8.4% during the past week (as measured by front-month contracts of WTI crude). Analysts expected further downward pressure from labor strikes in Kuwait, however, prices continued to climb, suggesting we may be on the other side of the bottom. WTI closed at $43.73 on Friday (see below).

oil_apr232016

Yields on U.S. treasuries of various durations climbed during the week. Two-year treasury bond yields were up to 0.84 percent, more than 13 percent above their previous week level. Ten-year treasury yields climbed more than seven period during the week. Corporate bond yields fell during the week, from the AAA level to the junk-bond level.

Equities were mixed on the week, with the Nasdaq down six-tenths of a percent following disappointing earnings at Google, and the Dow and S&P 500 up a hair more than half a percentage point, each.

twoten_apr232016

Exchange rates also continue to fluctuate from week-to-week. The pound sterling appreciated by more than a percent against the dollar, the Canadian dollar by more than two percent, and the Rand by more than 2.5 percent. Meanwhile, the Yen depreciated by more than 1.3 percent against the dollar during the one-week period. The Euro-dollar spot rate was unchanged on the week.

fx_apr232016

 

Dollar-indices: hitting recent lows (and how to plot with Python)

While the dollar has depreciated against most major currencies over the past month (notably 4.4% against the Canadian dollar, 3.3% against the Yen, 4.2% against the Real, and 8.6% against the Rand), the Fed H.10 release provides measures of exactly how much the value of the greenback has fallen relative to a weighted set of its trading partners. The latest data show that the U.S. dollar has fallen on April 12 to its lowest level since June 2015 compared to other major currencies, and its lowest level since October 2015 compared to a broad index of currencies.

This post covers the trade-weighted dollar and is split into two segments: a description of the index and its recent behavior, and a short python script showing how you can use the pandas and matplotlib libraries to retrieve the time series and plot it.

Two trade-weighted dollar indices, described

The Fed’s two most common trade-weighted indices of the foreign exchange value of the U.S. dollar are the major-currencies index and the broad-index. Both measure the value of the dollar relative to other countries’ currencies and intend to be aligned, through their weighting system, with the currencies closely related to U.S. trade patterns. The broad index, however, covers more currencies, and especially more emerging market currencies.

Discrepancies between even the broad index and the way the dollar is actually traded are inevitable. For example, the broad index weight in 2016 for Canada is 12.664 percent (broad index weights are updated yearly), whereas year-t0-date, 15.2 percent of total U.S. trade has been with Canada. From a share-of-total-U.S.-trade perspective, the broad index is somewhat overweight the Yuan, Euro, and Yen, and underweight the Canadian dollar and Mexican peso.

dtwexm_apr152016
The major currencies index contains fewer currencies than the broad index, and has fallen to a 22-month low. (plot from dashboard)

From 2005 through 2015, the major currencies index remained largely within a range of 70-85. After a steep climb through 2015, in January, the measure peaked at 95.8, to the frustration of U.S. exporters whose customers essentially pay more for the same goods (and as a result buy fewer). On April 12, the major currencies index hit its lowest level since June 2015.

Meanwhile, the broad index has moved in the same direction, hitting a five month low on April 12. The plot of the broad index is below, including how to obtain it. You can substitute DTWEXB with DTWEXM if you are interested in the major currencies index.

Python: Retrieve and plot the trade-weighted dollar

Next we show how Python can be used to gather and plot data on the Fed’s broad index of the foreign exchange value of the dollar. The script gathers data from Fred and plots each business day’s index value since the start of 2014.

Gathering data

First, we import pandas, numpy, and matplotlib and give them conventional short names. We will also use datetime and date.

In [1]:
# Import libraries
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import datetime
from datetime import date

Next, we use the pandas.io.data package to request the data from Fred. I’ve found the code for our series of interest, DTWEXB, by searching, but you can also find it on the Fred site by source (Board of Governors of the Federal Reserve System), or by release (H.10). We paste the series code into the datareader and provide start and end dates. Pandas retrieves the data into a dataframe.

In [2]:
import pandas_datareader.data as webdata
tstart = datetime.datetime(2014, 1, 1)
#retrieve trade-weighted dollar data from fred
dtwexb = webdata.DataReader("DTWEXB", "fred", tstart);
#display five most recent observations
dtwexb.tail(5)
Out[2]:
DTWEXB
DATE
2016-04-11 119.4527
2016-04-12 119.2860
2016-04-13 119.6278
2016-04-14 119.6472
2016-04-15 119.6701

When was the last time the measure was as low as its April 12 value?

In [3]:
print dtwexb[119.29>=dtwexb].dropna().tail(2)
              DTWEXB
DATE                
2015-10-21  119.1786
2016-04-12  119.2860

Line plot of data

Lastly, we can use matplotlib to plot the data.

In [4]:
#Create figure and plot dtwexb
fig = plt.figure(figsize=[7,5])
ax1 = plt.subplot(111)
line = dtwexb.DTWEXB.plot(color='blue',linewidth=2)

#Add a title
ax1.set_title('Trade-Weighted U.S. Dollar, Broad Index (1997=100)', fontsize=15)

#Add y label and no x-label since it is dates
ax1.set_ylabel('Index')
ax1.set_xlabel('')

#Axis options
ax1.spines["top"].set_visible(False)  
ax1.spines["right"].set_visible(False)  
ax1.get_xaxis().tick_bottom()
ax1.get_yaxis().tick_left()
ax1.tick_params(axis='x', which='major', labelsize=8)
ax1.yaxis.grid(True)

#Annotate with text
fig.text(0.15, 0.85,'Last: ' + str(dtwexb.DTWEXB[-1]) \
         + ' (as of: ' + str(dtwexb.index[-1].strftime('%Y-%m-%d'))\
         + ')');
url = 'https://research.stlouisfed.org/fred2/series/TWEXB'
fig.text(0.05, 0.025, 'Source: ' + url)
fig.text(0.65, 0.16, 'briandew.wordpress.com')

#Save as png
plt.savefig('dtwexb.png', dpi=1000)
dtwexb
The broad index hit a five-month low on April 12.

Dashboard update: lowered expectations

Dashboard PDF file:

Macro and Markets Dashboard: United States (April 16, 2016 — PDF)

Dashboard update summary:

Much of the past week’s macroeconomic news offered disappointment, yet markets dismissed the weak data as a result of what seems like lowered expectations. Retail sales, business inventories, and industrial production reports showed weakness in March (though labor market continues to look good). Corporate earnings have been quite soft in the first quarter, and GDP figures are likely to reflect the first quarter slowdown in output. Investors, however, seem relatively more risk-on, despite the weak macroeconomic data. Their expectations about earnings have fallen low enough to not only absorb the recent results, but to react positively in some cases. These investors are also faced with fewer high-earning alternatives, given sluggish and slowing growth abroad.

CPI and PPI data show little change, while oil prices continue to rebound. Currency markets are quite active, especially on the emerging markets side, where the dollar is weaker, partially as a result of the commodity price rebound.

Macro and Labor Market Indicators

Industrial production and total capacity utilization both fell in March. The industrial production index was down 0.6 points, largely stemming from decreases in production in mining and utilities market groups. This was magnified in the total percentage of capacity utilized figure, which fell to its lowest level since 2010. Mining capacity grew 1.6 percent year over year, despite a simultaneous 12.9 percent fall in production.

tcu_apr162016

The inventories to sales ratio ticked up in February, as a result of a fall in sales of 1.4 percent and an increase of inventories by 1.2 percent over their February 2015 levels. The most recent week’s data on new jobless claims showed the fewest new claims since 1973.

Equities and Fixed Income

Despite the weak data from the corporate side, equity markets were up and corporate bond yields down during the week. The Nasdaq and Dow were up 1.8 percent on the week, while the S&P 500 climbed 1.6 percent. Yields on U.S. Treasury bonds at all maturities, and U.S. corporate bonds at all maturities and credit ratings, have fallen over the past month, pushing prices higher.

Prices and Currency Markets

This week brought the release of March PPI and CPI data. Both were largely unchanged, as the PPI for all commodities increased slightly and the CPI fell slightly, over their February year over year percentage change levels. There was a surprise in CPI apparel prices, which fell 0.6 percent over their March 2015 levels, despite an increase in February. The data continue to reflect the very low prices of energy and moderate prices increase in healthcare and housing.

prices_apr162016

Currency markets were busy during the past week. The dollar strengthened against the Euro (0.91 percent), Yen (0.81 percent), and Swiss Franc (1.06 percent), while, as pointed out by FT’s hard currency, the dollar weakened further against the four R’s, the ruble, real, ringgit, and rand. These four currencies have seen a dramatic change in direction over the past month, appreciating against the dollar by more than 3.5 percent each and more than 5 percent in the case of the rand.

Note:

I try to gradually improve the dashboard and how I summarize changes. Any feedback would be much appreciated. Please feel free to leave a comment, or send me an email at brianwdew@gmail.com.

Dashboard update: signs of price pressure

Macro and Markets Dashboard: United States (April 9, 2016 — PDF)

The first full week of April saw active but net slightly down equity markets. While new economic data during the week was positive, expectations about corporate profits and output levels in the first quarter of 2016 are low. In light of solid fundamentals, and increasing aggregate demand, pessimists (including on the campaign trail) seem to be overreacting. Price data shows signs of upward pressure after an extended decline, and may soon join labor market and equity market indicators in signalling accelerating economic expansion.

The Nasdaq composite index fell 1.3 percent on the week, while both the S&P 500 and the Dow Jones Industrial Average fell 1.2 percent. Volatility, as measured by the VIX, closed Friday 17 percent above its previous week level. The Shiller index of price to earnings ratios climbed in March to 25.5. Expectations about first quarter earnings are very weak.

pe_apr092016

Prices data showed a continuation of upward pressure in March from commodity and food prices. Oil prices climbed more than eight percent during the past week. U.S. oil inventory fell for the first time in eight weeks. March CPI data, due out next week, should reflect the rising fuel and commodity prices.

wti_apr092016

World food prices, measured by the Food and Agriculture Organization of the UN, showed an uptick in March from a jump in sugar prices. This is only the second material increase in the world food price index since early 2014.

ffpi_apr092016

The trade-weighted dollar index continues to show a depreciation in the U.S. dollar. To the frustration of the Bank of Japan, the Yen appreciated nearly three percent against the dollar during the past week, which is not included in the lagged trade-weighted index. The dollar did appreciate against many emerging market currencies, pound sterling, and the Canadian dollar, during the past week.

fx_apr092016

Oil Price Data with Python

This example shows how Python can be used to take a look at oil prices. The script gathers daily oil price data from Quandl and plots how the price has changed over the past few months.

Gathering data

First, we import pandas, numpy, and matplotlib and give them conventional short names.

In [1]:
# Import libraries
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline

Next, we identify the url for our data. In this case, the data is provided by Quandl and the url can be obtained by clicking ‘csv’ under the API for any series on the right-hand side of the page. We read the CHRIS CME_CL1 csv file provided by Quandl into a pandas dataframe.

In [2]:
# Import from Quandl WTI crude oil price data
url = "https://www.quandl.com/api/v3/datasets/CHRIS/CME_CL1.csv"
wticl1 = pd.read_csv(url, index_col=0, parse_dates=True)
wticl1.sort_index(inplace=True)
wticl1_last = wticl1['Last']
wticl1['PctCh'] = wticl1.Last.pct_change()

Line plot of oil price

Lastly, we can use matplotlib to generate a line plot showing the most recent 68 days worth of closing prices for WTI crude front month contracts. The past week has seen this measure of oil prices reach nearly $40 per barrel.

In [3]:
fig = plt.figure(figsize=[7,5])
ax1 = plt.subplot(111)
line = wticl1_last.tail(68).plot(color='red',linewidth=3)
ax1.set_ylabel('USD per barrel')
ax1.set_xlabel('')
ax1.set_title('WTI Crude Oil Price', fontsize=18)
ax1.spines["top"].set_visible(False)  
ax1.spines["right"].set_visible(False)  
ax1.get_xaxis().tick_bottom()
ax1.get_yaxis().tick_left()
ax1.tick_params(axis='x', which='major', labelsize=8)
fig.text(0.15, 0.85,'Last: $' + str(wticl1.Last[-1])\
         + ' (as of: ' \
         + str(wticl1.index[-1].strftime('%Y-%m-%d'))\
         + ')');
fig.text(0.15, 0.80,'Change: $' + str(wticl1.Change[-1])\
         + '; ' \
         + str((np.round((wticl1.PctCh[-1] * 100), \
         decimals=2))) + '%')
fig.text(0.1, 0.06, 'Source: ' + url)
fig.text(0.1, 0.02, 'briandew.wordpress.com')
plt.savefig('oil.png', dpi=1000)
oil

Dashboard update: green shoots in March

Macro and Markets Dashboard: United States (April 2, 2016 — PDF)

The final week of March capped off a month of solid equity market gains and encouraging macroeconomic data. New data shows a continued strengthening of labor markets and a reduction in volatility. Bond yields fell during the week, and the dollar depreciated against most major currencies.

All three major U.S. equity market indices were up more than 6.5 percent on the month, while the Nasdaq composite index climbed nearly three percent during the week ending April 1. Volatility, as measured by the VIX, fell to its lowest level since August 2015. Indeed, nearly all major U.S. asset classes posted gains during the month, following a bearish January and February. West Texas Intermediate (WTI) crude oil prices jumped 13.6 percent in March, though they declined more than 2.5 percent during the most recent week.

manpmi_apr022016

Manufacturing Purchasing Managers Index (PMI) data for March from the Institute for Supply Management (ISM) suggests improvement in manufacturing conditions (see above chart). The PMI, which can be thought of as the weighted percentage of purchasing managers who report positively (above 50 suggests growth), posted its first increase in six months. As a bonus, this monthly report comes with possibly the most pithy explanation accompanying any statistic:

PMI® at 51.8%

New Orders and Production Growing
Employment and Inventories Contracting
Supplier Deliveries Slower

The U.S. economy added 215,000 jobs in March, while unemployment figures ticked up slightly to five percent. However, as evidenced in the dashboard and in previous posts, an increase in the labor force participation rate tells a more complex story than an increase in the unemployment rate. A strong labor market will attract people who are otherwise not participating (someone without a job but not looking for work is not considered unemployed under the headline unemployment figure from the BLS). The labor force participation rate has experienced its first six consecutive months without decline since 2005, as people are being drawn into a decent labor market.

unemp_apr022016

Across the board, U.S. bond yields fell during the past week. The real yield curve on a five-year U.S. treasury pushed negative, reaching further than a quarter point into the red (see below). The yield on a ten-year treasury fell to 1.79 percent on Friday, from 1.91 a week earlier. Corporate bond yields in all credit segments were also down during the one-week period.

fiveyearrealyield_apr022016

Touching on some additional data, personal and personal disposable income both increased by 0.2 percent in February. The personal savings rate ticked up to 5.4 percent in February, from 5.3 percent in January. As expected, the net international investment position of the United States continued to deteriorate in Q4 of 2015. The economic policy uncertainty monthly index fell nearly 22 percent in March, providing further evidence for a reduction in uncertainty-related volatility.

Lastly, over the past week the U.S. dollar depreciated against all major non-pegged currencies. Notably, the greenback weakened by more than two percent against the Swiss franc and Canadian dollar, more than one and a half percent against the Yen, and around one percent against the Yuan during the five-day period.

Machine reading IMF data with Python: an example

UPDATE June 16, 2016: The IMF API has changed. My updated tutorial is here:

https://briandew.wordpress.com/2016/05/01/machine-reading-imf-data-data-retrieval-with-python/


Download PDF Example – Machine Reading IMF Data with Python

During the past 18 months, the International Monetary Fund has made all of its data available free-of-charge, and developed an API to allow access to data. This means data from sources such as International Financial Statistics (IFS) can be collected without the need for manual download.

I’ve written a crude example using Python. The example retrieves IMF Direction of Trade Statistics to show the declining U.S. share of world exports and the rapid rise of China’s export-led economy.

imf_api-Copy1_26_0

If you are getting started with Python for economic research, I recommend the free version of the Enthought Canopy deployment, which integrates several useful tools.