{"id":714,"date":"2025-01-31T14:42:51","date_gmt":"2025-01-31T06:42:51","guid":{"rendered":"http:\/\/iamfisher.net\/fisher\/?page_id=714"},"modified":"2026-06-20T18:04:46","modified_gmt":"2026-06-20T10:04:46","slug":"r1-zero-%e5%92%8c-r1","status":"publish","type":"page","link":"https:\/\/iamfisher.net\/fisher\/%e5%a5%87%e5%bc%82\/r1-zero-%e5%92%8c-r1\/","title":{"rendered":"R1-Zero \u548c R1"},"content":{"rendered":"<p><!--more--><\/p>\n<p>\u6839\u636e\u73b0\u6709\u7684\u4fe1\u606f\u548c\u6280\u672f\u903b\u8f91\uff0c\u7f51\u7edc\u4e0a\u7684\u8bf4\u6cd5\u5728\u4ee5\u4e0b\u65b9\u9762\u662f\u5408\u7406\u7684\uff0c\u4f46\u9700\u6ce8\u610f\u51e0\u70b9\u5173\u952e\u7ec6\u8282\uff1a<\/p>\n<hr \/>\n<h3><strong>1. R1-Zero \u4e0e R1 \u7684\u6838\u5fc3\u533a\u522b<\/strong><\/h3>\n<ul>\n<li><strong>R1-Zero<\/strong>\uff1a\n<ul>\n<li><strong>\u5b8c\u5168\u4f9d\u8d56\u5f3a\u5316\u5b66\u4e60\uff08RL\uff09<\/strong>\uff1a\u8fd9\u4e00\u8bf4\u6cd5\u662f\u5408\u7406\u7684\u3002\u5f3a\u5316\u5b66\u4e60\u901a\u8fc7\u73af\u5883\u4ea4\u4e92\u548c\u5956\u52b1\u673a\u5236\u4f18\u5316\u6a21\u578b\u7b56\u7565\uff0c\u65e0\u9700\u6807\u6ce8\u6570\u636e\uff0c\u9002\u5408\u590d\u6742\u63a8\u7406\u4efb\u52a1\u3002<\/li>\n<li><strong>\u5f3a\u8c03\u63a8\u7406\u80fd\u529b<\/strong>\uff1aRL \u7684\u63a2\u7d22\u6027\u786e\u5b9e\u80fd\u63d0\u5347\u6a21\u578b\u7684\u81ea\u4e3b\u63a8\u7406\u80fd\u529b\uff0c\u5c24\u5176\u5728\u5f00\u653e\u6027\u95ee\u9898\u4e2d\uff08\u5982\u6570\u5b66\u8bc1\u660e\u3001\u7b56\u7565\u6e38\u620f\uff09\u3002<\/li>\n<li><strong>\u6f5c\u5728\u9650\u5236<\/strong>\uff1a\u5b8c\u5168\u4f9d\u8d56 RL \u53ef\u80fd\u9700\u8981\u66f4\u957f\u7684\u8bad\u7ec3\u65f6\u95f4\uff0c\u4e14\u521d\u671f\u8868\u73b0\u53ef\u80fd\u4e0d\u7a33\u5b9a\u3002<\/li>\n<\/ul>\n<\/li>\n<li><strong>R1<\/strong>\uff1a\n<ul>\n<li><strong>\u51b7\u542f\u52a8\u4f7f\u7528\u76d1\u7763\u5b66\u4e60<\/strong>\uff1a\u8fd9\u4e00\u8bbe\u8ba1\u7b26\u5408\u5b9e\u9645\u9700\u6c42\u3002\u76d1\u7763\u5b66\u4e60\u80fd\u5feb\u901f\u8d4b\u4e88\u6a21\u578b\u57fa\u7840\u8bed\u8a00\u80fd\u529b\uff08\u5982\u8bed\u6cd5\u3001\u5e38\u89c1\u8868\u8fbe\uff09\uff0c\u89e3\u51b3\u521d\u671f\u8f93\u51fa\u7684\u4e00\u81f4\u6027\u95ee\u9898\u3002<\/li>\n<li><strong>\u76d1\u7763\u6570\u636e\u4e0d\u63d0\u5347\u63a8\u7406\u80fd\u529b<\/strong>\uff1a\u76d1\u7763\u5b66\u4e60\u4e3b\u8981\u4f18\u5316\u8868\u5c42\u8bed\u8a00\u6a21\u5f0f\uff0c\u800c\u975e\u6df1\u5c42\u903b\u8f91\u63a8\u7406\uff0c\u56e0\u6b64\u8fd9\u4e00\u8bf4\u6cd5\u6210\u7acb\u3002<\/li>\n<li><strong>\u9002\u7528\u573a\u666f<\/strong>\uff1a\u9002\u5408\u9700\u8981\u5feb\u901f\u90e8\u7f72\u4e14\u5bf9\u8bed\u8a00\u89c4\u8303\u6027\u8981\u6c42\u9ad8\u7684\u573a\u666f\uff08\u5982\u5ba2\u670d\u3001\u5185\u5bb9\u751f\u6210\uff09\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr \/>\n<h3><strong>2. \u7f51\u7edc\u8bf4\u6cd5\u7684\u5408\u7406\u6027\u9a8c\u8bc1<\/strong><\/h3>\n<ul>\n<li><strong>\u6280\u672f\u53ef\u884c\u6027<\/strong>\uff1a\n<ul>\n<li><strong>\u76d1\u7763+RL \u7684\u6df7\u5408\u8bad\u7ec3<\/strong>\uff1a\u8fd9\u662f\u5f53\u524d\u5927\u6a21\u578b\u7684\u5e38\u89c1\u8303\u5f0f\uff08\u5982 ChatGPT \u521d\u671f\u7528\u76d1\u7763\u5fae\u8c03\uff0c\u540e\u7528 RLHF \u4f18\u5316\uff09\uff0c\u56e0\u6b64 R1 \u7684\u8bbe\u8ba1\u7b26\u5408\u884c\u4e1a\u5b9e\u8df5\u3002<\/li>\n<li><strong>\u7eaf RL \u6a21\u578b\u7684\u6311\u6218<\/strong>\uff1a\u5b8c\u5168\u4f9d\u8d56 RL \u9700\u8981\u9ad8\u6548\u7684\u5956\u52b1\u8bbe\u8ba1\u548c\u73af\u5883\u6a21\u62df\uff0c\u6280\u672f\u96be\u5ea6\u8f83\u9ad8\uff0c\u4f46\u5e76\u975e\u4e0d\u53ef\u80fd\uff08\u5982 AlphaGo Zero \u7684\u6210\u529f\u6848\u4f8b\uff09\u3002<\/li>\n<\/ul>\n<\/li>\n<li><strong>\u672f\u8bed\u51c6\u786e\u6027<\/strong>\uff1a\n<ul>\n<li><strong>\u201c\u4e0d\u9700\u8981\u76d1\u7763\u6570\u636e\u201d<\/strong>\uff1a\u4e25\u683c\u6765\u8bf4\uff0cRL \u4ecd\u9700\u5956\u52b1\u4fe1\u53f7\uff08\u53ef\u89c6\u4e3a\u9690\u5f0f\u76d1\u7763\uff09\uff0c\u4f46\u65e0\u9700\u4f20\u7edf\u6807\u6ce8\u6570\u636e\uff0c\u56e0\u6b64\u8bf4\u6cd5\u57fa\u672c\u6b63\u786e\u3002<\/li>\n<li><strong>\u201c\u51b7\u542f\u52a8\u201d<\/strong>\uff1a\u6307\u6a21\u578b\u4ece\u96f6\u5f00\u59cb\u8bad\u7ec3\u65f6\u7684\u521d\u59cb\u5316\u9636\u6bb5\uff0c\u4f7f\u7528\u76d1\u7763\u5b66\u4e60\u52a0\u901f\u6536\u655b\u662f\u5408\u7406\u7b56\u7565\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr \/>\n<h3><strong>3. \u9700\u8c28\u614e\u5bf9\u5f85\u7684\u6f5c\u5728\u8bef\u533a<\/strong><\/h3>\n<ul>\n<li><strong>\u201cR1-Zero \u548c R1 \u662f\u5b8c\u5168\u4e0d\u540c\u7684\u4ea7\u54c1\u201d<\/strong>\uff1a\n<ul>\n<li>\u66f4\u53ef\u80fd\u4e3a<strong>\u540c\u4e00\u6846\u67b6\u4e0b\u7684\u4e0d\u540c\u914d\u7f6e<\/strong>\uff0c\u800c\u975e\u72ec\u7acb\u4ea7\u54c1\u3002\u4f8b\u5982\uff0cR1-Zero \u53ef\u80fd\u662f R1 \u7684\u201c\u65e0\u76d1\u7763\u53d8\u4f53\u201d\uff0c\u7c7b\u4f3c GPT-3 \u4e0e GPT-3.5 \u7684\u5173\u7cfb\u3002<\/li>\n<\/ul>\n<\/li>\n<li><strong>\u201c\u76d1\u7763\u5b66\u4e60\u4ec5\u7528\u4e8e\u8bed\u8a00\u4e00\u81f4\u6027\u201d<\/strong>\uff1a\n<ul>\n<li>\u76d1\u7763\u6570\u636e\u53ef\u80fd\u9690\u542b\u90e8\u5206\u63a8\u7406\u6a21\u5f0f\uff08\u5982\u6570\u5b66\u9898\u7684\u6807\u51c6\u89e3\u6cd5\uff09\uff0c\u95f4\u63a5\u5f71\u54cd\u63a8\u7406\u80fd\u529b\uff0c\u4f46\u4e3b\u8981\u76ee\u6807\u4ecd\u662f\u8bed\u8a00\u89c4\u8303\u6027\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr \/>\n<h3><strong>\u7ed3\u8bba<\/strong><\/h3>\n<p>\u7f51\u7edc\u4e0a\u7684\u8bf4\u6cd5<strong>\u5728\u6280\u672f\u903b\u8f91\u548c\u884c\u4e1a\u5b9e\u8df5\u4e0a\u662f\u5408\u7406\u7684<\/strong>\uff0c\u4f46\u9700\u6ce8\u610f\uff1a<br \/>\n1. R1-Zero \u548c R1 \u66f4\u53ef\u80fd\u662f\u540c\u4e00\u6280\u672f\u8def\u7ebf\u7684\u4e0d\u540c\u9636\u6bb5\u6216\u914d\u7f6e\uff0c\u800c\u975e\u5b8c\u5168\u72ec\u7acb\u7684\u4ea7\u54c1\u3002<br \/>\n2. \u76d1\u7763\u5b66\u4e60\u5bf9\u63a8\u7406\u80fd\u529b\u7684\u95f4\u63a5\u5f71\u54cd\u53ef\u80fd\u88ab\u4f4e\u4f30\uff0c\u4f46\u5176\u4e3b\u8981\u4f5c\u7528\u786e\u5b9e\u662f\u8bed\u8a00\u89c4\u8303\u5316\u3002<br \/>\n3. \u7eaf RL \u6a21\u578b\u7684\u8bad\u7ec3\u96be\u5ea6\u548c\u8d44\u6e90\u6d88\u8017\u8f83\u9ad8\uff0c\u5b9e\u9645\u5e94\u7528\u4e2d\u53ef\u80fd\u4ecd\u9700\u6df7\u5408\u65b9\u6cd5\u3002<\/p>\n<p>\u5982\u9700\u8fdb\u4e00\u6b65\u9a8c\u8bc1\uff0c\u5efa\u8bae\u67e5\u9605\u5b98\u65b9\u6280\u672f\u6587\u6863\u6216\u8bba\u6587\uff0c\u660e\u786e\u6a21\u578b\u7684\u5177\u4f53\u67b6\u6784\u548c\u8bad\u7ec3\u7ec6\u8282\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"parent":9,"menu_order":0,"comment_status":"open","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-714","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/pages\/714","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/comments?post=714"}],"version-history":[{"count":0,"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/pages\/714\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/pages\/9"}],"wp:attachment":[{"href":"https:\/\/iamfisher.net\/fisher\/wp-json\/wp\/v2\/media?parent=714"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}