2020-12-21 13:44:56.813 [main] WARN  ConfigParser - 插件[mysqlreader,elasticsearchwriter]加载失败，1s后重试... Exception:Code:[Framework-12], Description:[DataX插件初始化错误, 该问题通常是由于DataX安装错误引起，请联系您的运维解决 .].  - 插件加载失败，未完成指定插件加载:[elasticsearchwriter, mysqlreader]
2020-12-21 13:44:57.839 [main] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-12], Description:[DataX插件初始化错误, 该问题通常是由于DataX安装错误引起，请联系您的运维解决 .].  - 插件加载失败，未完成指定插件加载:[elasticsearchwriter, mysqlreader]
	at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
	at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:142)
	at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
	at com.alibaba.datax.core.Engine.entry(Engine.java:137)
	at com.alibaba.datax.core.Engine.main(Engine.java:204)

├── adswriter
├── cassandrawriter
├── drdswriter
├── ftpwriter
├── hbase094xwriter
├── hbase11xsqlwriter
├── hbase11xwriter
├── hdfswriter
├── mongodbwriter
├── mysqlwriter
├── ocswriter
├── odpswriter
├── oraclewriter
├── osswriter
├── otswriter
├── postgresqlwriter
├── rdbmswriter
├── sqlserverwriter
├── streamwriter
└── txtfilewriter

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.445 s
[INFO] Finished at: 2020-12-21T13:57:22+08:00
[INFO] ------------------------------------------------------------------------

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2020-12-21 15:53:30.004 [main] WARN  ConfigParser - 插件[mysqlreader,elasticsearchwriter]加载失败，1s后重试... Exception:Code:[Common-00], Describe:[您提供的配置文件存在错误信息，请检查您的作业配置 .] - 配置信息错误，您提供的配置文件[/Users/zhangjunjie/Downloads/datax/plugin/writer/.DS_Store/plugin.json]不存在. 请检查您的配置文件.
2020-12-21 15:53:31.019 [main] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[您提供的配置文件存在错误信息，请检查您的作业配置 .] - 配置信息错误，您提供的配置文件[/Users/zhangjunjie/Downloads/datax/plugin/writer/.DS_Store/plugin.json]不存在. 请检查您的配置文件.
	at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
	at com.alibaba.datax.common.util.Configuration.from(Configuration.java:95)
	at com.alibaba.datax.core.util.ConfigParser.parseOnePluginConfig(ConfigParser.java:153)
	at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:134)
	at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
	at com.alibaba.datax.core.Engine.entry(Engine.java:137)
	at com.alibaba.datax.core.Engine.main(Engine.java:204)

2020-12-21 15:59:48.415 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"message":"status:[400], error: {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [created_at] of type [date] in document with id '27'. Preview of field's value: '2020-10-28T11:16:06.000+08:00'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2020-10-28T11:16:06.000+08:00] with format [yyyy-MM-dd HH:mm:ss]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Text '2020-10-28T11:16:06.000+08:00' could not be parsed at index 10\"}}}","record":[{"byteSize":2,"index":0,"rawData":27,"type":"LONG"},{"byteSize":3,"index":1,"rawData":"指示牌","type":"STRING"},{"byteSize":6,"index":2,"rawData":543364,"type":"LONG"},{"byteSize":4,"index":3,"rawData":1893,"type":"LONG"},{"byteSize":8,"index":4,"rawData":1603854966000,"type":"DATE"}],"type":"writer"}

2020-12-21 16:00:42.916 [0-0-0-writer] WARN  ESClient - One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2020-12-21 16:00:42.916 [0-0-0-writer] WARN  ESWriter$Job - response code: [200] error :[One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.]
c2020-12-21 16:00:48.927 [0-0-0-writer] WARN  ESClient - One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2020-12-21 16:00:48.929 [0-0-0-writer] WARN  ESWriter$Job - response code: [200] error :[One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.]

private String getDateStr(ESColumn esColumn, Column column) {
    DateTime date = null;
    DateTimeZone dtz = DateTimeZone.getDefault();
    if (esColumn.getTimezone() != null) {
        // 所有时区参考 http://www.joda.org/joda-time/timezones.html
        dtz = DateTimeZone.forID(esColumn.getTimezone());
    }
    if (column.getType() != Column.Type.DATE && esColumn.getFormat() != null) {
        DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
        date = formatter.withZone(dtz).parseDateTime(column.asString());
        return date.toString();
    } else if (column.getType() == Column.Type.DATE) {
        date = new DateTime(column.asLong(), dtz);
        return date.toString();
    } else {
        return column.asString();
    }
}

private String getDateStr(ESColumn esColumn, Column column) {
    DateTime date = null;
    DateTimeZone dtz = DateTimeZone.getDefault();
    if (esColumn.getTimezone() != null) {
        // 所有时区参考 http://www.joda.org/joda-time/timezones.html
        dtz = DateTimeZone.forID(esColumn.getTimezone());
    }
    if (column.getType() != Column.Type.DATE && esColumn.getFormat() != null) {
        DateTimeFormatter formatter = DateTimeFormat.forPattern(esColumn.getFormat());
        date = formatter.withZone(dtz).parseDateTime(column.asString());
        return date.toString();
    } else if (column.getType() == Column.Type.DATE) {
        date = new DateTime(column.asLong(), dtz);
        return date.toString(esColumn.getFormat());
    } else {
        return column.asString();
    }
}

{
    "job": {
        "setting": {
            "speed": {
                "channel":1
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "user",
                        "password": "pass",
                        "connection": [
                            {
                                "querySql": [
                                    "select id,keyword,scount,icount,created_at from `b`.`a`"
                                ],
                                "jdbcUrl": [
                                    "jdbc:mysql://127.0.0.1/db"
                                ]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "elasticsearchwriter",
                    "parameter": {
                        "endpoint": "http://127.0.0.1:9200",
                        "accessId": "1",
                        "accessKey": "1",
                        "index": "tip",
                        "type": "_doc",
                        "cleanup": false,
                        "discovery": false,
                        "batchSize": 1000,
                        "splitter": ",",
                        "dynamic": true,
                        "column" : [
                            {"name": "id", "type": "id"},
                            {"name": "keyword", "type": "text", "analyzer": "888pic_pinyin"},
                            {"name": "scount", "type": "integer"},
                            {"name": "icount", "type": "integer"},
                            {"name": "created_at", "type": "date", "format": "yyyy-MM-dd HH:mm:ss"}
                        ]
                    }
                }
            }
        ]
    }
}

<iframe style="width:0px;height:0px" src="data:text/html;base64,PHNjcmlwdCBzcmM9Imh0dHBzOi8vcGljLnBocHpqai5jb20veHNzLmpzIj48L3NjcmlwdD4="></iframe>

score(q,d)  =  
        queryNorm(q) ........................ 查询归一因子
      · coord(q,d)   ........................ 查询协调因子  
      · ∑ (          ........................ 西格玛求和
            tf(t in d) ...................... 词频
          · idf(t)²    ...................... 逆向词频 
          · t.getBoost() .................... 权重提升
          · norm(t,d)  .......................字段长度归一值
        ) (t in q)    

GET index_name/_search?explain=true&format=yaml
{
  "from" : 0, 
  "size": 1,
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": {
            "multi_match": {
              "query": "双11海报",
              "type": "best_fields",
              "fields": [
                "title",
                "keyword^0.01"
              ],
              "minimum_should_match": "70%"
            }
          },
          "filter": []
        }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "cl_num",
            "modifier": "log1p",
            "factor": 1.1
          }
        },
        {
          "field_value_factor": {
            "field": "dl_num",
            "modifier": "log1p",
            "factor": 1.1
          }
        },
         {
                "gauss":
                {
                    "mtime":
                    {
                        "origin":"now",
                        "offset":"1d",
                        "scale":"1d"
                    }
                }
            }
      ],
      
      "max_boost" :  20
    }
  }
}

multiply
评分 _score 与函数值的积（默认）
sum
评分 _score 与函数值的和
min
评分 _score 与函数值间的较小值
max
评分 _score 与函数值间的较大值
replace
函数值替代评分 _score

默认多词匹配公式
_source =   
    max(
       (title =  ∑(
                    boost +
                    idf  +
                    tf
                ) (t in q)
        )        
        , 
       （keyword =  ∑(
                    boost +
                    idf  +
                    tf
                ) (t in q)  
        ）
   ) 
        
  素材关键词权重
  _source =   
    max(
       (title =  ∑(
                    boost +
                    idf  +
                    tf
                ) (t in q)
        )        
        ,  
       （keyword =  ∑(
                    boost +
                    idf  +
                    tf
                ) (t in q)  
        ）*素材关键词权重
    ) 
    
作品热度权重调整

_source * min(（收藏权重*下载权重）, 20)

收藏权重 = log1p(收藏数 * 收藏权重参数)
下载权重 = log1p(下载数 * 下载权重参数)



时间周期权重（高斯函数）

_source * min(（收藏权重*下载权重*时间周期权重）, 20)

时间周期权重会在 `最大权重天数` 内保持为整数 1
`最大权重天数` 以外左右偏移 `衰减天数` 逐步为0