Agri Market Data

Overview

The Agmarknet data collected by the Ministry of Agriculture in India reports daily price and arrival data at a market/mandi level for 344 commodities across more than 650 districts of India. Paired with a dynamic system that not only updates this list of commodities but also adds on to the 4000 markets currently being covered (in response to geographical and administrative changes), makes it one of the most comprehensive wholesale price and quantity datasets. As of 31st December 2021, it stands at 54+ million price and 49+ million arrival data points reported over a span of 21 years.

CEDA’s Agmarknet tool helps a user to visualise and interact with this data in a graphical and GIS format while also allowing them to download the raw files for individual research. Moreover, the tool’s repository is populated with the latest data every month to maintain an up-to-date image of the Agmarknet portal.

Dataset Variables

• price

  Name	Data Type	Unit/Format	Comments
  date	string	YYYY-MM-DD	Start Year: 2001
  state_name	string	-	As per Agmarknet
  state_id	string	-	As per Agmarknet (Unique)
  district_name	string	-	As per Agmarknet
  district_id	integer	-	As per Agmarknet (Unique within State)
  market_name	string	-	As per Agmarknet
  market_id	integer	-	As per Agmarknet (Unique)
  census_state_name	string	-	As per Census 2011
  census_state_id	integer	-	Coded by CEDA (Unique)
  census_district_name	string	-	As per Census 2011
  census_district_id	integer	-	Coded by CEDA (Unique)
  commodity_name	string	-	As per Agmarknet
  commodity_id	integer	-	As per Agmarknet (Unique)
  variety	string	-	Commodity Variety
  grade	string	-	Commodity Grade
  min_price	numeric	Rs. per Quintal	Minimum Price
  max_price	numeric	Rs. per Quintal	Maximum Price
  modal_price	numeric	Rs. per Quintal	Modal Price

• arrival

  Name	Data Type	Unit/Format	Comments
  date	string	YYYY-MM-DD	Start Year: 2000
  state_name	string	-	As per Agmarknet
  state_id	string	-	As per Agmarknet (Unique)
  district_name	string	-	As per Agmarknet
  district_id	integer	-	As per Agmarknet (Unique within State)
  market_name	string	-	As per Agmarknet
  market_id	integer	-	As per Agmarknet (Unique)
  census_state_name	string	-	As per Census 2011
  census_state_id	integer	-	Coded by CEDA (Unique)
  census_district_name	string	-	As per Census 2011
  census_district_id	integer	-	Coded by CEDA (Unique)
  commodity_name	string	-	As per Agmarknet
  commodity_id	integer	-	As per Agmarknet (Unique)
  quantity	numeric	Tonnes	Arrival Quantity

Processing and Cleaning

Repeated Commodities

Agmarknet’s backend uses unique commodity IDs for storing and querying data. However, some commodities have been listed with two separate IDs causing a division of data (can be seen in the commodity dropdown menu on their Home Page). Our pipeline merges these repeats to ensure consistency and collate all respective data within one commodity ID.

Repeated Markets

Agmarknet’s backend uses unique market IDs for storing and querying data. Some markets however have been listed with two separate IDs — either within the same district or across two districts of the same state — causing a division of data. Our pipeline merges these repeats to ensure consistency and collate all respective data within one market ID.

Handling Duplicates

The price dataset is cleaned to maintain uniqueness at the date, market, commodity, commodity variety, commodity grade level, i.e. there are no two differing price values for a particular commodity of a certain variety and grade at a market on a given date. On the other hand, the arrival dataset maintains uniqueness at the date, market, commodity level by summing up arrival values if applicable, i.e. if a market reports two arrival quantities for a commodity on a given data we simply add them up.

This step is performed in the end to handle repeated market and commodity data. For instance, if the same price is listed for both Coconut IDs in a market, we drop the duplicate.

Census District Mapping

For uniformity and comparison with other modules of CEDA’s Data Portal, we have mapped all Agmarknet districts and markets to the 640 districts as reported in Census 2011. This has been done by first attempting a direct fuzzy match on district names and then resolving discrepancies by ascertaining each market’s current geolocation and matching it to previous administrative borders. Specifically, all matches fall in one of the following 6 reasons:

• 1: Direct match (Agmarknet has a district Mumbai, Maharashtra and so does Census 2011).
• 2: Spelling or name variant (Mohali, Punjab is listed as SAS Nagar, Punjab in Census 2011).
• 3: State formations (Adilabad, Telangana in Agmarknet corresponds to Adilabad, Andhra Pradesh of Census 2011).
• 4: District splits (Balod, Chattisgarh a district in Agmarknet was split from Durg, Chattisgarh in 2012).
• 5: Change/confusion in district HQ (Mangaldoi, Assam is a separate district according to Agmarknet but listed as the HQ/capital of Darrang, Assam as per Census 2011).
• 6: Miscellaneous needing market-level geomapping.

In matches of type 6, we jump to the market level and find each market in an agmarknet district that needs to be shifted to a different Census 2011 district. This involves both removing and adding markets by investigating which district would they lie in if we were looking at a 2011 map.

*Researchers can ascertain our market-level mapping from the raw data available for download, which includes both the Agmarknet State, District, Market and the corresponding Census 2011 district we have matched it to.
*We periodically update this mapping as and when Agmarknet adds new districts or markets.

Inconsistent Raw Updates

In some cases, Agmarknet moves a location (district/market) to either a different existing parent location or a completely new state/district. For instance, market1 of district1 is shifted under district2. Or market1 of district1 is made into a separate district itself. In such situations, the processed data of market1 remains consistently mapped to the same 2011 census district it belonged to before. The raw data on the other hand will variate, i.e. the data pre-update will stay under the original state/district and the data post-update will be recorded under the new state/district. To help keep track of such instances, the following table is regularly updated:

Time Series Feature

• prices: At the district level, for the given month range and commodity, we average over daily Modal Prices across all the markets of selected Census districts and return one data
  point per month for a district. Similarly, at the State and all India levels, an average over daily Modal Prices across all the markets for that commodity in the State and across all India in a month is plotted, respectively.
• arrivals: At the district level, for the given month range and commodity, we add up the daily Arrival Quantity across all the markets of selected Census districts and return one
  data point per month for a district. Similarly, at the State and all India levels, the sum of daily arrivals across all the markets for that commodity in the State and across all India in a month is plotted, respectively.

GIS/Map Feature

• prices: For the given month range and commodity, we average over daily Modal Prices across all the markets of each district (according to Census 2011) in India and return one data point per month for a district.
• arrivals: For the given month range and commodity, we add up the daily Arrival Quantity across all the markets of each district (according to Census 2011) in India and return one data point per month for a district.

Raw Dataset Download

Users can download raw data (same variables as mentioned before) from our website by following these steps:

1. State Selection
   • This queries from the census_state_id variable and returns Census 2011 mapped data. So for instance, to look up price data in Telangana you will need to select Andhra Pradesh in the dropdown menu.
   • However, the dataset will also contain the original (Agmarknet) market, district and state for every row to allow researchers to perform their own custom mapping and analysis.
2. Commodity Selection
   • This queries from the commodity_id variable which means it will return data for all varieties and grades of the commodity.
   • We allow multiple and unrestricted commodity selections via a checkbox feature.
3. Year Selection
   • Given the size of the Agmarknet data, we have restricted period selection to 1 year.
   • Price data begins from 2001 and Arrival data begins from 2000.
   • Data of the latest month will only be available on the 15th of every successive month to ensure a lag for any updates on previous data, i.e. January 2022 data will be live on our website after 15th February 2022.

Format: CSV
Updated Upto: 28th February 2022

Commodity List

Above list updated on 1st March 2022.

Acknowledgements: Tanish Bafna, an undergraduate student at Ashoka and an intern with CEDA during 2021-22, led the initiative to develop the Daily Food Prices tool.