Using the IMF Data API: Retrieving an IFS series with Python

April 5, 2017 Update: I’ve put together a newer version of the code/guide located here.

The following tutorial shows how to use python’s requests package to get data directly from the International Monetary Fund (IMF). The IMF’s application programming interface (API) provides access to economic and financial data of more than 180 countries over more than 60 years.

More information on the IMF Data API

The IMF offers guidance on using their data services. The API is not stable, so check the IMF data services news if you receive error messages.

More information on Python

This example works with the Anaconda distribution of Python 2.7.

A short example: Loading IMF data into pandas

Below is a short working example of loading the Australian export price index time-series from International Financial Statistics (IFS) into a pandas dataframe directly from the IMF API.

In [1]:

# Example: loading IMF data into pandas

# Import libraries
import requests
import pandas as pd

# URL for the IMF JSON Restful Web Service, 
# IFS database, and Australian export prices series
url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/IFS/Q.AU.PXP_IX.?startPeriod=1957&endPeriod=2016'

# Get data from the above URL using the requests package
data = requests.get(url).json()

# Load data into a pandas dataframe
auxp = pd.DataFrame(data['CompactData']['DataSet']['Series']['Obs'])

# Show the last five observiations
auxp.tail()

Out [1]:

@BASE_YEAR @OBS_VALUE @TIME_PERIOD
232 2010 94.6044171093095 2015-Q1
233 2010 90.4668716801789 2015-Q2
234 2010 90.4668716801789 2015-Q3
235 2010 85.5465473860777 2015-Q4
236 2010 81.5208275090858 2016-Q1

Breaking down the URL

The ‘key’ in our request is the URL, which contains instructions about which data we want.

The URL has three parts,

  1. http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/ The base for requests using the JSON restful data service;
  2. IFS/ the database ID, IFS, for International Financial Statistics;
  3. Q.AU.PXP_IX.?startPeriod=1957&endPeriod=2016 the data dimension and time period information.

The third part, data dimension and time period information, is broken down on the IMF Web Service knowledge base as:

{item1 from dimension1}+{item2 from dimension1}{item N from dimension1}.{item1 from dimension2} +{item2 from dimension2}+{item M from dimension2}? startPeriod={start date}&endPeriod={end date}

For guidance on finding dimension information and building your request, see my previous example of using the IMF API to retrieve Direction of Trade Statistics (DOTS) data. 

Cleaning up the dataframe

Let’s add more meaningful headers and set the date as our index

In [2]:

# Rename columns
auxp.columns = ['baseyear','auxp','date']

# Set the price index series as a float (rather than string)
auxp.auxp = auxp.auxp.astype(float)

# Read the dates as quarters and set as the dataframe index
rng = pd.date_range(pd.to_datetime(auxp.date[0]), periods=len(auxp.index), freq='QS')
auxp = auxp.set_index(pd.DatetimeIndex(rng))
del auxp['date']

# Show last five rows
auxp.tail()

Out [2]:

baseyear auxp
2015-01-01 2010 94.604417
2015-04-01 2010 90.466872
2015-07-01 2010 90.466872
2015-10-01 2010 85.546547
2016-01-01 2010 81.520828

Plot the data

Now we can use matplotlib to create a line plot showing the history of the Australian export price index.

In [3]:

# import matplotlib and pyplot
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline

# Create line plot and add labels and title
auxp.auxp.plot(grid=True, figsize=(9, 5), color="orange", linewidth=2,)
plt.ylabel('Index')
plt.xlabel('Year')
plt.title('Australia: Export Price Index (baseyear=' + str(auxp.baseyear[0]) + ')');

Out [3]:

Output from script

Save as a csv file

To save the data for use in another program, you can easily create a csv file.

In [4]:

auxp.to_csv('auxp.csv')
Advertisements

Brexit and the feedback of uncertainty through asset prices

The medium- and long-run consequences of Brexit are unknown, but increased uncertainty in response to the event will likely have real global effects of its own. Individuals are now less confident in their guesses about the future, and this uncertainty changes behavior, increasing risk aversion and decreasing business investment. Global markets reaction to Brexit reflects not only the new average belief about expected future earnings, but also the indirect feedback effects of the higher level of uncertainty.

Medium- and long-term effects unclear

The U.K.’s June 23 vote to leave the European Union, termed ‘Brexit’, surprised many analysts. As the referendum vote neared, markets (representing the average belief of individuals) increasingly expected the ‘remain’ party to win. Some well-educated analysts explained how Brexit would never happen (possibly what Nassim Nicholas Taleb would consider proof of a black swan). The ‘leave’ campaign won with 51.9 percent of the vote, prompting David Cameron’s announced resignation.

The medium-term consequences for the U.K. and E.U. (or anywhere else) are entirely unclear. For example, should unemployment increase and production falter in the U.K. during the next few months, the Bank of England may need to ease monetary policy further and adopt very unconventional tools. However, if the market reaction to Brexit can be fully absorbed by exchange rate adjustment, the Bank of England may need to raise interest rates in response.

On June 24, the pound sterling fell more than eight percent against the dollar (figure 1). In response to the currency depreciation, many here in the U.S. joked about buying goods online from the U.K. or planning vacations. Many others expressed uncertainty about the future of the U.K. and the E.U., and the economic and political implications for the U.S.

gbp_jun262016
Figure 1. The pound sterling fell against most currencies following the Brexit vote, including by more than eight percent in one day against the U.S. dollar.

Markets reminded of their anxiety issues

Within hours of the ‘leave’ campaign victory, stock prices of some financial services companies, such as Lloyds, fell by 30 percent. Although real changes from the vote may take two years to implement, the average belief about the appropriate cost of equity for these firms changed dramatically overnight. The fundamentals for people and businesses in the U.K. do not immediately change, but their behavior and asset prices do, and this has an indirect feedback effect on the fundamentals

Likewise, the U.S. is clearly not part of the E.U. or U.K., however, it is intimately linked to both markets through trade and investment. The Dow Jones Industrial Average fell 610 points (3.39%) on Friday in response to the news. The most widely-used measure of market volatility jumped nearly 50 percent (figure 2). The sales revenue of the largest U.S. companies is practically unchanged, however, individuals’ collective behavior has changed in response to uncertainty.

Individuals are repricing the assets that they previously overvalued. However, if the Bank of England is forced to wait to decide which direction its key policy will turn, how well can individuals immediately reprice future earnings? The immediate reaction to Brexit reflects not a new certainty about future earnings but increased uncertainty. Individuals never know what the future will hold, but these large unexpected events make them less confident in all of their guesses, and it changes their behavior.

vix_jun252016
Figure 2. VIX, the CBOE measure of expected near-term equity market volatility, saw an almost 50 percent daily increase following the Brexit result

Searching for safety and the uncertainty feedback loop

When people are less certain about the future, they take fewer risks. For example, business investment falls when equity market volatility is high. Individual and institutional investors also show preference for safe assets during times of increased uncertainty. In an immediate reaction to Brexit, U.S. treasuries, gold, and Japanese Yen saw inflows, while equity markets saw net outflows. Neither business fundamentals nor any rules had changed; people were showing risk aversion.

Investors’ increased aversion to equity investment raises firms’ equity cost of capital. This cost of capital is a major factor in firms’ investment decisions, especially for smaller and less-cash-flush firms. Therefore, the uncertainty induced asset price shock has a feedback mechanism through which it affects the future value of companies. This dangerous feedback mechanism can be procyclical.

What it means for the U.S.

The medium- and long-term direct effects of Brexit on the U.S. are very unclear. Brexit-induced equity market volatility, higher levels of uncertainty, and the negative effects these entail can however be analyzed with attention to the current U.S. macroeconomic environment. Three potential short-run consequences emerge:

1) Continued volatility in equity markets and strong demand for treasuries;

2) More downward pressure on business investment, which was already negative in the first quarter of 2016; and

3) Delay of the next Federal Funds rate hike in response to the above.

While the downward pressure on already weak business investment is worrisome, none of the above are enough to induce major concern. Households, many of which have recently started seeing long-awaited wage and income increases, will play a key role in determining whether recent asset price volatility will spill into consumer sentiment and awake a much worse set of feedback mechanisms.

Check out the dashboard and please leave feedback

Nearly 100 freshly updated charts:

U.S. Macro and Markets Dashboard (Updated June 26, 2016)

References and additional reading

Networks, Crowds, and Markets: Reasoning About a Highly Connected World by David Easley and Jon Kleinberg

Nick Bloom comment on Brexit 

BBC: The UK’s EU referendum: All you need to know

 

Dashboard update: bond yields fall with renewed demand for safe assets and lower interest rate expectations

Monitor more than 80 economic indicators with the macro and markets dashboard:

dash_open
United States Macroeconomic and Markets Dashboard: Updated June 11, 2016

Dashboard weekly update summary:

The latest labor market data show continued overall improvement in wages and low levels of new jobless claims, offering some consolation after the surprisingly weak May jobs report (see last week’s update). Equity market volatility increased, however, as already lackluster global growth forecasts were revised down by the World Bank. Domestic and foreign investors are shifting their portfolio of assets to fixed streams of income. Global bond yields, including on U.S. government and corporate debt, fell considerably during the past week’s rally. Investors are searching for higher returns on safe assets and responding to lower interest rate expectations.

Wages grow faster than productivity

Narrowing a long and persistent gap between productivity and wage growth, recent data suggest wages have been increasing in many industries. For most of the past decade, worker’s productivity (the output for each hour of work) grew more rapidly than their wages. The gap was in part from technology making work more efficient, but it also came from a weak labor market. An economy in which there are many qualified workers for each opening makes workers less likely to quit and more likely to accept no or small increases in wages. Companies simply do not need to rely on pay increases as a motivation when fear of unemployment is very strong.

While wage growth crawled along for a decade, productivity growth remained strong. Recent data suggest, however, that the long upward trend in productivity may be facing at least a hiccup. Meanwhile, overall measures show wages have been growing at a reasonable pace since mid-2014. Revised first quarter 2016 index data on wages and productivity shows the former nearly catching the latter.

wageout_jun112016

When oil prices fall, for example, there is a transfer of wealth from the stakeholders of oil producers (who face a fall in revenue) to households (who spend less on fuel and energy). Likewise, a fall or stagnation in corporate profits can result in an increase in worker’s relative share of output. Wages, unlike commodity or stock prices, tend not to be cut. Where the past year has seen labor markets and workers’ bargaining power improve, it has seen productivity and corporate profits stagnate.

lshare_jun112016

Additionally, data for the week of June 4 on new claims of joblessness was strong, with only 264,000 such claims. Overall, reasonable wage increases, an increasing labor share of output, and low headline unemployment paint a better picture for households and aggregate demand than the last jobs report’s payroll growth and participation rate data suggest. That said, labor market improvements are traditionally slow and gradual, while deterioration is rapid and steep. The June jobs report should therefore have major implications for the Fed’s rate hike decision.

Equity market volatility climbs

U.S. equity markets gains over the first four trading days of the week were erased on Friday by a large sell off. The CBOE volatility index, VIX, increased to 17 from 13.5 a week earlier. The bond rally described below suggests that there has been a flight to safety. Investors have been adjusting in part to new forecasts of generally lower global growth. Likewise, there are several large events on the horizon (brexit, elections, central bank policy divergence, etc.) suggesting fluctuations and jumps in equity markets (as well as debt and forex markets).

vix_jun112016

Bond markets rally as investors seek safe assets and interest rate expectations fall

Over the past week, U.S. treasury bonds, t-bills, and corporate bond yields fell. Elsewhere, ten-year German bund yields are nearly negative and Japanese ten-year government bond yields have fallen to -0.13%. People’s tolerance for extremely low returns is limited, and U.S. government debt offers a relatively higher return. During the past week, ten-year U.S. treasury bonds reached a four year low, partially as a result of strong foreign demand.

yield_jun112016

The spread between 10-year and 2-year treasuries sits currently at a nine-year low. Potential causes for the flat yield curve include the following: 1) investor search for return driving driving down long- and medium-duration bond yields, 2) investor fear of a business cycle downturn and a near future need for monetary easing, and 3) lower interest rate expectations as a result of recent data taking June (and potentially July) rate hikes off the table.

spread_jun112016

Check out the full dashboard for more than 80 indicators of U.S. economic activity:

U.S. Macroeconomic and Markets Dashboard, June 11, 2016

I also updated the dashboard for Japan:

Japan Macroeconomic and Markets Dashboard, June 11, 2016

 

 

U.S. economy added surprisingly few jobs in May, likely delaying the next Fed funds rate hike

Monitor more than 80 economic indicators with the macro and markets dashboard:

dash_open
United States Macroeconomic and Markets Dashboard: Updated June 4, 2016

Dashboard update summary:

The employment summary released on Friday was surprisingly weak, with only 38,000 new jobs added to the U.S. economy during May and previous month figures revised down. The BLS estimate was dramatically below consensus forecasts. Many macroeconomic indicators remain positive and in line with Fed targets; households have been spending and investing more and inflation is above one percent. However, the surprise job growth weakness is sufficient to delay the expected timetable for Fed interest rate hikes.

payrolls_jun052016

Macro: GDP revised upward but business investment still strongly negative

In the second estimate, 2016 Q1 U.S. real gross domestic product (GDP) growth was revised upward to 0.8 percent from 0.5 percent. Private inventory investment did not decrease as much as previously estimated. Gross domestic investment from businesses remains strongly negative, following negative corporate profits in 2015 Q4 and very low profits in 2016 Q1. Slow real GDP growth is a result of the offset of negative business investment on sound household data. Household incomes increased in April, while expenditures decreased slightly in real terms but remain strong.

gdi_jun052016

Jobs day disappoints

Economists’ consensus view that between 100,000 to 200,000 jobs were added to the U.S. economy in May was met on Friday with a surprisingly paltry 38,000 increase estimate from the BLS. The weak data was surprising enough to cause large immediate jumps in bond and foreign exchange markets.

Meanwhile, the headline unemployment rate fell to 4.7 percent from 5.0 percent, the largest decrease in several months. The fall in the headline rate had more to do with people “leaving the labor force” than with the addition of new jobs, unfortunately. The labor force participation rate fell to 62.6 percent in May. Many of those who left the labor force had been unemployed for 27 weeks or more and where therefore discouraged enough to stop trying to find a job.

civpart_jun052016

Bond and FX markets react

Treasury bill and bond constant maturity yields fell in response to the expected delay of interest rate hikes. The yield curve (see below a simple visualization) remains relatively low and flat. The real yield on a five year treasury, for example, fell to -0.22 percent on Friday, June 3. Foreign exchange markets saw an almost universally stronger U.S. dollar in response to the jobs report. The dollar closed roughly two percent weaker against the Euro, Yen, Australian Dollar, and Ruble, roughly one and a half percent weaker against the Swiss Franc, Brazilian Real, Turkish Lira, and Malaysian Ringgit, and more than three percent against the South African Rand.

yc_jun052016

fx_jun052016

Rate hike delay

While U.S. macroeconomic data has not been thrilling, labor market strength and wage increases were, in the previous few months, supportive of a June Fed Funds target rate increase of a quarter point, to 0.5-0.75 percent. The new jobs report, therefore, softens the strongest pillar. Markets have basically taken a June hike off the table. The next jobs report will be watched very closely and will likely determine whether a July hike is appropriate. Given the timing of Fed meetings and the expected gradual increase rate, it is more likely that we only see one rate hike during 2016.

See more than 80 indicators in the Dashboard:

dash_open

Dashboard update: Fed minutes and recent data advance interest rate hike expectations

Monitor more than 80 economic indicators with the macro and markets dashboard:

dash_open
United States Macroeconomic and Markets Dashboard: Updated May 21, 2016

Dashboard update summary:

New data continue to strengthen the U.S. macroeconomic picture. Minutes from the most recent meeting of the Federal Reserve Open Markets Committee (FOMC) signal that the U.S. central bank may increase its key federal funds interest rate target by 25 basis points in June. Markets have responded by increasingly pricing higher interest rate expectations into the bond and foreign exchange markets.

Industrial production stronger than expected

The Federal Reserve publishes an index of U.S. industrial production that goes back to 1919. The monthly index data for April was released on Tuesday, and showed greater-than-expected industrial sector output. This was a result largely of a month-over-month spike in production in the utilities sector, which had previously been hard hit by declines in both demand for utilities and commodity prices (see below). The index had declined each month so far during the year, and rebounded by seven tenths of a percent from March to April.

ip_may212016

Jobless claims drop to less alarming level

Data from the week of May 7 on the number of new claims of joblessness hit a recent high, drawing some attention. New data for the week of May 14 were better, with 278,000 new jobless claims during that week.

Inflation low and steady but pointed higher

April data on inflation, as measured by the consumer price index, was released on Tuesday. Annualized inflation for all items was 1.1 percent in April, while the Core CPI (the CPI excluding food & beverage, and energy) grew by 2.1 percent. Healthcare continues to grow at the fastest rate of any major item category in the basket. Energy and transportation costs are still below their previous year level, but less so than in March (see below).

cpi_may212016

FOMC minutes suggest June hike more likely

On Wednesday, the Fed released minutes from the FOMC meeting at the end of April. The minutes indicate that if new labor market data shows continued strengthening, and inflation continues toward two percent, then the committee will likely raise the federal funds target rate in June.

fomc_may212016

An increase in the federal funds rate causes other short-term (and to a lesser extent long-term) interest rates to rise. In essence, the fed funds target rate should act as a floor on the cost of risk-free borrowing of U.S. dollars.

Reaction of markets

Expectations about future interest rates play a major role in finance. Short term interest rate increases, like those from fed funds rate increases, generate broad effects, including on long term interest rates, the demand for money in the real economy, the propensity to save and invest, and bond and foreign exchange markets. Markets react today to changes today in expectations about the future. This happened following the release of the FOMC minutes; U.S. treasury bond prices fell immediately. The yield on ten- and two-year treasury bonds jumped, closing the week at 1.85% and 0.89% respectively.

tyield_may212016

See more indicators, as well as foreign exchange rates, in the dashboard:

dash_open

Dashboard update: Jobs data and new uncertainty

Dashboard PDF file:

Macro and Markets Dashboard: United States (May 7, 2016 — PDF)
Dashboard Update Summary:

Jobs data for April showed payrolls continue to grow, but at a slower rate. Wage data was strong, however, the labor force participation rate gave up much of its recent improvement. Uncertainty surrounding markets and economic policy seems to have increased in the recent week, and fewer economists now predict a Fed rate hike in June. U.S. equity markets were down for the second consecutive week, while corporate bond yields rose and treasury yields fell. Recent data showed improvement in the trade balance from the weaker dollar, however, the recent depreciation trend has also become less certain.

Jobs Report showed slower jobs growth but wage improvement

The U.S. added 160,000 jobs in April, compared with 208,000 in March and 233,000 in February (both previous months were also revised downward). By sector, much of the growth came from the services side, on an annualized basis. Construction jobs, which make up less than five percent of nonfarm payrolls, were up 4.1 percent, while mining and logging jobs continued their decline and are now down more than fifteen percent over the past year (this is the smallest industry sector shown in the figure below, and represents only 0.4 percent of nonfarm payrolls). Weekly data on new jobless claims, as of April 30, showed still very low, but slightly increased, levels.

jobsector_may072016

The latest jobs report shows continued improvement in both nominal and real wages in practically all sectors. Nominal wages increased most rapidly over the past year in financial services, information services, and leisure and hospitality. On average, wages from the goods sector are higher, largely as a result of low-wage service-sector jobs in leisure and hospitality.

wages_may072016

Equity and Bond market conditions deteriorated

Equity markets were down for the second straight week. The S&P 500 was down 0.4 percent, the Nasdaq composite index was down 0.8 percent, and the Dow Jones industrial average was 0.2 percent lower. Volatility was higher during the week, and the VIX closed Friday at 14.7. The Shiller index of price to earnings ratios was up to 26.02 percent in April from 25.54 in March. Corporate bond yields ticked up during the week. The Merrill Lynch index of junk bond yields was up to 7.56 percent. Ten year treasury yields fell to 1.79 percent.

Economic policy uncertainty improved in April but may revert

Economic policy uncertainty, as measured by Baker, Bloom, and Davis, fell sharply in April, as there was little speculation of Fed action at the April meeting. However, I expect this index to bounce back; uncertainty will increase as the Fed June meeting and Brexit grow closer.

epu_may072016

Oil was down on the week, while April food prices increased

Oil prices closed lower on the week. The U.S. measure of crude oil prices, West Texas Intermediate crude front-month contracts, fell 2.7 percent during the week, to $44.66 a barrel. World food prices from the Food and Agriculture Organization (which I half-jokingly also use as a proxy of political instability) ticked up slightly in April, but remain low.

A weaker dollar improved the trade balance in March

The Fed’s trade-weighted dollar broad index against major currencies fell last Friday (April 29–past week data is released on Mondays) to its lowest level since May 2015. The year-to-date rapid depreciation of the dollar has cut import quantities, as further evidenced in the March data on trade. The trade deficit, which remains roughly 2.2 percent of GDP, improved to -40.4B in March. However, more recent foreign exchange data shows uncertainty about recent depreciation trends. The dollar was stronger against nearly all major trading partners during the past week, notably 1.2 percent against the British pound, 2.76 percent against the Canadian dollar, 3.16 percent against the Australian dollar, 4.5 percent against the Turkish lira, 3,86 percent against the Mexican peso, and 4.3 percent against the South African rand.

tb_may072016

Machine Reading IMF Data: Data Retrieval with Python 

Updated November 16, 2016

Introduction

The International Monetary Fund (IMF) Statistics Department (STA) allows API access to their economic time series. Well-known datasets such as International Financial Statistics (IFS) can be machine read through the API. This example will use Python to retrieve Direction of Trade Statistics (DOTS) data from STA’s JSON RESTful Web Service so that we can determine the United States’ share of world exports over the past 50 years.

The IMF knowledge base provides more information on the three avaiable API formats and IMF data services. For more information on the work of STA, see their PDF annual report (PDF), STA at a glance 2015.

Gathering series and dimension information

First, we will need to import the requests and pandas libraries. These will allow us to read json data, open urls, and request information from the web.

Libraries

In [1]:
# Import libraries
import requests
import pandas as pd

Since we are using the JSON RESTful API, we start by using the ‘Dataflow’ endpoint URL to look at what series are available and find the series id of interest. The full output is long, so I’ve removed the data unrelated to this example. The IMF has many more series than what is shown below.

Find Series Name

In [2]:
# Find the series id and text name.
url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/Dataflow/'
seriesids = requests.get(url).json()
df = pd.DataFrame(seriesids['Structure']['Dataflows']['Dataflow'])
for x in range(6, 13):
     items = (str(df['@id'][x]), str(df['Name'][x]['#text']))
     print ': '.join(items)
DOT: Direction of Trade Statistics (DOTS)
FSIREM: Financial Soundness Indicators (FSI), Reporting Entities - Multidimensional
CDIS: Coordinated Direct Investment Survey (CDIS)
GFS01M: Government Finance Statistics (GFS 2001) - Multidimensional
GFS01: Government Finance Statistics (GFS 2001)
BOP: Balance of Payments (BOP)
BOPAGG: Balance of Payments (BOP), World and Regional Aggregates

We found above that the id for Direction of Trade Statistics is DOT. We can use this id to read notes about the series. We will next need to identify the dimensions of the data. For example, direction of trade data is based on a home country a flow and measure and a counterpart country. The data also has multiple frequencies and units of measurement. All of this information will be needed to later make our data request.

Find Series Details and Description

In [3]:
# Annotations for the series
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/DataStructure/DOT"
dotstruct = requests.get(url).json()
df = pd.DataFrame(dotstruct['Structure']['KeyFamilies']\
     ['KeyFamily']['Annotations'])
for x in range(0, 7): 
     items = (str(df['Annotation'][x]['AnnotationTitle']), \
     str(df['Annotation'][x]['AnnotationText']['#text']))
     print ': '.join(items)
Latest Update Date: 04/26/2016
Name: Direction of Trade Statistics (DOTS)
Temporal Coverage: Monthly and quarterly data are available starting 1960. Annual data are available starting 1947.
Geographic Coverage: DOTS covers 184 countries, the world, and major areas.
Methodology: Guide to Direction of Trade Statistics, 1993. See Documents tab.
Definition: The <B>Direction of Trade Statistics (DOTS)</B> presents current figures on the value of merchandise exports and imports disaggregated according to a country's primary trading partners. Area and world aggregates are included in the display of trade flows between major areas of the world. Reported data is supplemented by estimates whenever such data is not available or current. Imports are reported on a cost, insurance and freight (CIF) basis and exports are reported on a free on board (FOB) basis, with the exception of a few countries for which imports are also available FOB. Time series data includes estimates derived from reports of partner countries for non-reporting and slow-reporting countries.
Code: DOT

Find Series Dimensions

In [4]:
# Look at structure of DOTS data to find the dimensions for our data request
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/DataStructure/DOT"
dotstruct = requests.get(url).json()
df = pd.DataFrame(dotstruct['Structure']['KeyFamilies']['KeyFamily']\
          ['Components']['Dimension'])
for x in range(0, 4): 
     items = ("Dimension", str(x+1), str(df['@codelist'][x]))
     print ': '.join(items)
Dimension: 1: CL_FREQ
Dimension: 2: CL_AREA_DOT
Dimension: 3: CL_INDICATOR_DOT
Dimension: 4: CL_COUNTERPART_AREA_DOT

We can now copy the code for each dimension into the CodeList Method to get the list of possible values. For example, we will need to identify the value of the second dimension, CL_AREA_DOT, for the United States. Below, we show that the code is US. I’ve manually placed the index number for the U.S. and World codes (again to save space), however, you can replace [200, 248] with [0, 248] to get the full list of country/area codes.

Find Country Codes

In [5]:
# Obtain country codes
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/CodeList/CL_AREA_DOT"
country = requests.get(url).json()
df = pd.DataFrame(country['Structure']['CodeLists']['CodeList']['Code'])
for x in [200, 248]: 
     items = (str(df['@value'][x]), str(df['Description'][x]['#text']))
     print ': '.join(items)
US: United States
W00: All Countries, excluding the IO

The series ID is DOT and the country codes (we will use this with the exporting country, CL_AREA_DOT, and the counterpart, CL_COUNTERPART_AREA_DOT) of interest are W00 for world and US for the US. We see below that the indicator of interest is TXG_FOB_USD, Goods, Value of Exports, Free on board (FOB), US Dollars.

Find Column IDs

In [6]:
# Obtain series info and ids
url = "http://dataservices.imf.org/REST/SDMX_JSON.svc/CodeList/CL_INDICATOR_DOT"
series = requests.get(url).json()
df = pd.DataFrame(series['Structure']['CodeLists']['CodeList']['Code'])
for x in range(0, 4): 
     items = (str(df['@value'][x]), str(df['Description'][x]['#text']))
     print ': '.join(items)
TXG_FOB_USD: Goods, Value of Exports, Free on board (FOB), US Dollars
TMG_CIF_USD: Goods, Value of Imports, Cost, Insurance, Freight (CIF), US Dollars
TMG_FOB_USD: Goods, Value of Imports, Free on board (FOB), US Dollars
All Indicators: All Indicators

We repeat the above steps for each dimension and record which series values are of interest to us.

Retrieving Data

The guide to STA’s API shows how we can combine information from the previous steps to call and retrieve data. For direction of trade statistics, we see that the dimensions are as follows:

  • Dimension 1: CL_FREQ (the frequency of the data–we want to use monthly data) – M
  • Dimension 2: CL_AREA_DOT (the primary country) – US
  • Dimension 3: CL_INDICATOR_DOT (the measure–we want to look at exports free of board) – TXG_FOB_USD
  • Dimension 4: CL_COUNTERPART_AREA_DOT (the counterpart country) – W00

The JSON RESTful API method for requesting the data is the CompactData Method. The format for putting together dimension and time period information is shown on the Web Service knowledge base as:

http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/{database ID}/ {item1 from dimension1}+{item2 from dimension1}{item N from dimension1}.{item1 from dimension2} +{item2 from dimension2}+{item M from dimension2}? startPeriod={start date}&endPeriod={end date}

Putting all of this information together, the URL to retrieve a JSON dictionary for 1966-2016 US exports to the world data is:

http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US.TXG_FOB_USD.W00.?startPeriod=1981&endPeriod=2016

The python code which gets the data and saves it as a dictionary is as follows:

Request data from IMF API

In [7]:
url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US.TXG_FOB_USD.W00.?startPeriod=1981&endPeriod=2016'
data = requests.get(url).json()
usexp = pd.DataFrame(data['CompactData']['DataSet']['Series']['Obs'])
usexp.columns = ['status', 'usexports','date'];
usexp.tail()
Out[7]:
date usexports
415 2015-08 123065777734
416 2015-09 125394024247
417 2015-10 130599515853
418 2015-11 120731632371
419 2015-12 119907169367

We can repeat the above code with a different URL to obtain data on total world exports and the exports of other countries which we may want to compare with the United States. We combine the request for several series into one URL, by adding ‘+code2+code3’. For example, ‘US+JP+CN.TXG..’

Example of request with multiple columns

In [8]:
ourl = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/DOT/M.US+CN+JP+W00.TXG_FOB_USD.W00.?startPeriod=1972&endPeriod=2016'
odata = requests.get(ourl).json();

Cleaning the dataframe and naming rows

In [9]:
wexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][0]['Obs'])
wexp.columns = ['status','wexports','date']
del wexp['date']
del wexp['status']
chexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][1]['Obs'])
chexp.columns = ['status', 'chexports','date']
del chexp['date']
del chexp['status']
jpexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][2]['Obs'])
jpexp.columns = ['jpexports','date']
del jpexp['date']
usexp = pd.DataFrame(odata['CompactData']['DataSet']['Series'][3]['Obs'])
usexp.columns = ['status', 'usexports','date']
del usexp['status'];

Now we combine the two series into one dataframe and tell our script to read the export value columns as numbers.

Read as numeric

In [10]:
combined = pd.concat([usexp, wexp, chexp, jpexp], axis=1)
pd.to_datetime(combined.date)
combined = combined.set_index(pd.DatetimeIndex(combined['date']))
usexports = pd.to_numeric(combined.usexports)
wexports = pd.to_numeric(combined.wexports)
cexports = pd.to_numeric(combined.chexports)
jexports = pd.to_numeric(combined.jpexports)

Finally, we can calculate the U.S. percentage share of world exports. We simply divide the us exports by the world exports and multiply by 100. If using the data for economic research, we would likely take the log forms and apply some filters.

Capture

Calculate share of world exports for each country

In [11]:
combined['usshare'] = usexports / wexports * 100
combined['chinashare'] = cexports / wexports * 100
combined['japanshare'] = jexports / wexports * 100
combined.tail()

Out[11]:

date usshare chinashare japanshare
2015-08 9.460 15.121 3.668
2015-09 8.830 14.455 3.754
2015-10 9.330 13.737 3.892
2015-11 9.018 14.663 3.645
2015-12 8.776 16.362 3.804

Graphing the data

Let’s use matplotlib to view the result of our work.

Graph of US share of world exports

In [12]:
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
txt = '''Source: International Monetary Fund.'''

# Plot US share of world exports
combined.usshare.plot(grid=True, figsize=(9, 5), color="blue", linewidth=2,)
plt.ylabel('percentage of world exports')
plt.xlabel('Year')
plt.text(20,4.5,txt)
plt.title('U.S. Share of World Exports');
download

The graph shows a decrease in the U.S. share of exports from nearly 20 percent in 1966 to roughly 9 percent in 2015. We can also easily examine how changes in the U.S. share of exports compare with changes in the share of Japan and China.

Graph of moving average of US, China, and Japan shares of world exports

In [13]:
# Calculate moving average for each share, to reduce noise
combined['ussharema'] = combined['usshare'].rolling(12,12).mean()
combined['chsharema'] = combined['chinashare'].rolling(12,12).mean()
combined['jpsharema'] = combined['japanshare'].rolling(12,12).mean()

combshares = combined[['ussharema', 'chsharema', 'jpsharema']]
shares = list(combshares);
# Plot various shares of world exports
combined[shares][120:].plot(grid=True, figsize=(9, 5), linewidth=2)
plt.ylabel('percentage of world exports')
plt.xlabel('Year')
plt.text(150,-2,txt)
plt.title('Share of World Exports', );
download (1)

Export dataset to .csv

Let’s save the dataset in a portable format that can be read by any statistical software. My preference is to create a .csv file, which I will use for my U.S. Macroeconomic and Markets Dashboard.

Create csv file

In [14]:
combined.to_csv('us_share_exports.csv')

 

Dashboard update: Inaction has reactions

Dashboard PDF file:

Macro and Markets Dashboard: United States (April 30, 2016 — PDF)

Dashboard update summary:

Markets closed down slightly on the week following inaction from both the Fed and Bank of Japan. Oil and Yen both still became five percent more expensive in dollar terms. Advance estimate 2016 Q1 real GDP growth was weak at 0.6 percent, but in line with expectations. Labor markets continue to be a bright spot, with eyes on next Friday’s April jobs report.

rgdpgrowth_apr302016

Real GDP ticked up 0.6 percent in log terms in the first quarter of 2016. Personal consumption expenditures and residential fixed investment helped to keep GDP growth positive despite a decrease in nonresidential fixed and inventory investment, and growth in the trade gap. The personal savings rate also increased slightly, in quarterly terms, to 5.2 percent in Q1 from 5.0 percent the previous quarter.

FOMC meeting statement changes are kindly highlighted by the Wall Street Journal and include removal of the language about global risks and, some suggestion in my view, based on labor market growth, household income, and consumer sentiment, of a June rate hike. This would depend on continued labor market strength and price pressure plus an upward revision of Q1 GDP.

pcecomp_apr302016

Personal consumption expenditures (PCE) continued to increase in Q1, led by higher spending on services. However, monthly data on PCE as a share of GDP decreased slightly in March over its February level. PCE on durable goods as a percentage of GDP was also down slightly in March, to 7.3 percent from 7.4 percent in February.

Labor market data continues to be spotless. The weekly 257,000 new jobless claims is still near the multi-decade low. Next Friday is jobs day. Both the labor force participation rate and wages should continue to improve.

nasdaq_apr302016

Disappointing earnings data from Apple includes the first decline in iPhone sales nearly since its introduction and a slowdown in sales in China. This hit the Nasdaq particularly hard, as the composite index was down 2.7 percent while the S&P 500 and Dow Jones industrial average fell 1.3 percent each during the week.

oil_apr302016

Oil prices still managed to climb five percent over the past five days. West Texas Intermediate (WTI) crude oil was trading above $46 a barrel at several points during the week. Two recent stories reminded me how oil price fluctuations cause enormous transfers of wealth between countries. Jamaica was praised in this week’s Alphachat series on sovereign debt, while noting that their recent fiscal fortune is aligned with lower prices on their fuel imports. Likewise, a recent IMF publication noted that oil exporting economies in the middle east and central Asia have enacted powerful fiscal stimulus measures to keep their economies moving while they suffer the oil revenue slowdown. Those who believe in the resource curse might note that government measures to shift the economy away from oil are both important and difficult to achieve.

Lastly, the Yen had a volatile week, closing with a five percent appreciation against the dollar. The Brazilian real appreciated nearly four percent during the week, the Swiss franc and Turkish lira appreciated nearly two percent each, the pound sterling nearly one and a half percent and the Canadian dollar nearly one percent.

The full dashboard is here: Macro and Markets Dashboard: United States (April 30, 2016 — PDF)

Dashboard update: oil rally

Dashboard PDF file:

Macro and Markets Dashboard: United States (April 23, 2016 — PDF)

Dashboard update summary:

Oil prices continue to play a dominant role in other markets’ movements. As mentioned in previous updates, corporate earnings expectations are depressed, so even dismal performance (GS Q1 earnings fell 60% for example) can lead individual stocks higher. However, recovery in energy and commodity markets pushes investors from uncertainty-safe investments into riskier pools. This past week was relatively quiet on the macroeconomic data side, and market volatility was low, however, next week offers both Q1 GDP first estimates and an FOMC meeting, and volatility may tick back up.

While Doha talks set expectations for a fall in oil prices, markets actually pushed prices up by 8.4% during the past week (as measured by front-month contracts of WTI crude). Analysts expected further downward pressure from labor strikes in Kuwait, however, prices continued to climb, suggesting we may be on the other side of the bottom. WTI closed at $43.73 on Friday (see below).

oil_apr232016

Yields on U.S. treasuries of various durations climbed during the week. Two-year treasury bond yields were up to 0.84 percent, more than 13 percent above their previous week level. Ten-year treasury yields climbed more than seven period during the week. Corporate bond yields fell during the week, from the AAA level to the junk-bond level.

Equities were mixed on the week, with the Nasdaq down six-tenths of a percent following disappointing earnings at Google, and the Dow and S&P 500 up a hair more than half a percentage point, each.

twoten_apr232016

Exchange rates also continue to fluctuate from week-to-week. The pound sterling appreciated by more than a percent against the dollar, the Canadian dollar by more than two percent, and the Rand by more than 2.5 percent. Meanwhile, the Yen depreciated by more than 1.3 percent against the dollar during the one-week period. The Euro-dollar spot rate was unchanged on the week.

fx_apr232016

 

Dollar-indices: hitting recent lows (and how to plot with Python)

While the dollar has depreciated against most major currencies over the past month (notably 4.4% against the Canadian dollar, 3.3% against the Yen, 4.2% against the Real, and 8.6% against the Rand), the Fed H.10 release provides measures of exactly how much the value of the greenback has fallen relative to a weighted set of its trading partners. The latest data show that the U.S. dollar has fallen on April 12 to its lowest level since June 2015 compared to other major currencies, and its lowest level since October 2015 compared to a broad index of currencies.

This post covers the trade-weighted dollar and is split into two segments: a description of the index and its recent behavior, and a short python script showing how you can use the pandas and matplotlib libraries to retrieve the time series and plot it.

Two trade-weighted dollar indices, described

The Fed’s two most common trade-weighted indices of the foreign exchange value of the U.S. dollar are the major-currencies index and the broad-index. Both measure the value of the dollar relative to other countries’ currencies and intend to be aligned, through their weighting system, with the currencies closely related to U.S. trade patterns. The broad index, however, covers more currencies, and especially more emerging market currencies.

Discrepancies between even the broad index and the way the dollar is actually traded are inevitable. For example, the broad index weight in 2016 for Canada is 12.664 percent (broad index weights are updated yearly), whereas year-t0-date, 15.2 percent of total U.S. trade has been with Canada. From a share-of-total-U.S.-trade perspective, the broad index is somewhat overweight the Yuan, Euro, and Yen, and underweight the Canadian dollar and Mexican peso.

dtwexm_apr152016
The major currencies index contains fewer currencies than the broad index, and has fallen to a 22-month low. (plot from dashboard)

From 2005 through 2015, the major currencies index remained largely within a range of 70-85. After a steep climb through 2015, in January, the measure peaked at 95.8, to the frustration of U.S. exporters whose customers essentially pay more for the same goods (and as a result buy fewer). On April 12, the major currencies index hit its lowest level since June 2015.

Meanwhile, the broad index has moved in the same direction, hitting a five month low on April 12. The plot of the broad index is below, including how to obtain it. You can substitute DTWEXB with DTWEXM if you are interested in the major currencies index.

Python: Retrieve and plot the trade-weighted dollar

Next we show how Python can be used to gather and plot data on the Fed’s broad index of the foreign exchange value of the dollar. The script gathers data from Fred and plots each business day’s index value since the start of 2014.

Gathering data

First, we import pandas, numpy, and matplotlib and give them conventional short names. We will also use datetime and date.

In [1]:
# Import libraries
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import datetime
from datetime import date

Next, we use the pandas.io.data package to request the data from Fred. I’ve found the code for our series of interest, DTWEXB, by searching, but you can also find it on the Fred site by source (Board of Governors of the Federal Reserve System), or by release (H.10). We paste the series code into the datareader and provide start and end dates. Pandas retrieves the data into a dataframe.

In [2]:
import pandas_datareader.data as webdata
tstart = datetime.datetime(2014, 1, 1)
#retrieve trade-weighted dollar data from fred
dtwexb = webdata.DataReader("DTWEXB", "fred", tstart);
#display five most recent observations
dtwexb.tail(5)
Out[2]:
DTWEXB
DATE
2016-04-11 119.4527
2016-04-12 119.2860
2016-04-13 119.6278
2016-04-14 119.6472
2016-04-15 119.6701

When was the last time the measure was as low as its April 12 value?

In [3]:
print dtwexb[119.29>=dtwexb].dropna().tail(2)
              DTWEXB
DATE                
2015-10-21  119.1786
2016-04-12  119.2860

Line plot of data

Lastly, we can use matplotlib to plot the data.

In [4]:
#Create figure and plot dtwexb
fig = plt.figure(figsize=[7,5])
ax1 = plt.subplot(111)
line = dtwexb.DTWEXB.plot(color='blue',linewidth=2)

#Add a title
ax1.set_title('Trade-Weighted U.S. Dollar, Broad Index (1997=100)', fontsize=15)

#Add y label and no x-label since it is dates
ax1.set_ylabel('Index')
ax1.set_xlabel('')

#Axis options
ax1.spines["top"].set_visible(False)  
ax1.spines["right"].set_visible(False)  
ax1.get_xaxis().tick_bottom()
ax1.get_yaxis().tick_left()
ax1.tick_params(axis='x', which='major', labelsize=8)
ax1.yaxis.grid(True)

#Annotate with text
fig.text(0.15, 0.85,'Last: ' + str(dtwexb.DTWEXB[-1]) \
         + ' (as of: ' + str(dtwexb.index[-1].strftime('%Y-%m-%d'))\
         + ')');
url = 'https://research.stlouisfed.org/fred2/series/TWEXB'
fig.text(0.05, 0.025, 'Source: ' + url)
fig.text(0.65, 0.16, 'briandew.wordpress.com')

#Save as png
plt.savefig('dtwexb.png', dpi=1000)
dtwexb
The broad index hit a five-month low on April 12.